Dominik Werder :: Paul Scherrer Institut

File writer and data streaming

WP5 meeting, 2017-01-31, DMSC Copenhagen

File writer design

• Internal structure of the file writer
• Command interface towards the ECP
• Start file writing, e.g. by ECP
• Specify what is being written
• Get information about ongoing file writing
• Find the data that should be written
• Based on documents:
• BrightnESS NeXus Writer
• NeXusFileWriterDesign.md
• File writer flow diagram

File writer class hierarchy

File writer internal structure

Modular design, cover concerns:

• StreamReader/Writer
• Knows about flatbuffer schema
• Extract sourcename, timestamp from messages
• Write HDF datasets according to data in message
• Source
• Create StreamReader/Writer based on incoming data
• Interface for sourcename
• Demux
• Dispatch messages to Sources on muxed topics

File writer internal structure

• Streamer, StreamMaster
• Open and handle data topics / partitions
• Find start of data, seek in topics
• Feed data to demuxers
• FileWriterTask
• File handle, static data, list of Sources, Demuxers
• CommandListener
• Listen for commands on topic
• Setup file write task with list of streams, static data..
• Instantiate StreamMaster and hands off task

File writer command interface

• Commands: json, schema validated
• Static data provided by ECP
• "Metadata"
• User information, file name, ...
• Streaming data set up by ECP
• Topic
• Source name where applicable (muxed topics)
• Data type is inferred from received message
• Start write, optional start time
• Stop write, optional stop time
• Switch file: Start new file, do not miss any data

Find data to be written

Issues to solve:

• Find begin of data in the Kafka stream
• Find the last known value for slowly changing sources

Facts:

• Kafka log is not ordered in time
• Begin of data can possibly already exist in log at the time of ECP file write request, or not
• Last known value could be far away in the log

Find data to be written

Start file writing:

• Search first message of streams in partitions (index)
• Upper bound on latency allows to abort the search
• Caveat: Writer congestion, bad time sync

Stop/Switch file writing at given time:

• Stop if (message or EOF) long after stop time
• Upper bound allows to decide when to stop waiting
• Requires collaboration from producers:
  Sync. Latency mainly given by producer buffering

File writer monitors current latency with synchronized ESS time

Write last known value

• For slowly changing values, write last known value
• Motor positions, temperature values, ...
• No guaranteed upper bound on maximum age
• Could be already purged from the log
• Relevant on low-traffic topics only
• Many possible solutions, e.g.
• Backwards search (✔)
• Kafka using log compaction (fut ✔)
• Redis cache

Search backwards

• Open topic, seek backwards
• Extract last known values for data streams
• If not found for all, rewind further
• Worst case: reads the whole log on disk to find nothing

Log compaction

• Log cleaner purges only keys for which younger messages exist
• Kafka docs:

Log compaction

• Connect to broker 1
• Open replicated but compacted topic
• Read topic until end
• Extract youngest message for key
• Connect to broker 2
• Open topic, seek to where we left broker 1
• Continue with live data from there

Write last known value

• Which solution to choose?
• Search backwards
• Works regardless of broker infrastructure
• Log compaction
• Just another Kafka broker
• Redis cache: Same functionality, more infrastructure
• Same functionality as compaction
• Adds more infrastructure
• ECP should decide which streams get last known value

File writer status

• Simple file writing works full round-trip
• Initiated by a json command message
• Receive from data topics
• Create a simple HDF file
• Automatic basic round-trip test

Wire format

All messages must provide:

• Timestamp of message production
• Equals timestamp of data for most event based sources
• Let's us infer message production ideally to ± some hundreds ms
• Set as CreateTime / LogAppendTime, to allow index
• Source name (string) to differentiate sources on muxed topic

Wire format

• Flat buffer scheme currently identified by 16 bit
• Flexible, allows to incorporate other formats later
• Adds complexity if we only use flat buffers
• Proposal: switch to flatbuffer's file_identifier
• 4 byte identifier at a fixed location at the beginning
• Specified in schema definition:

  file_identifier = "abcd";

Round trip time

• Vary msg/s on x-axis, keep total bandwidth constant
• 20MB/s, 30MB/s, 40MB/s
• Max buffer 20 ms at producer
• 1 GB/s link: For large M, RTT given by link capacity

Broker CPU usage

• 20MB/s, 30MB/s, 40MB/s
• About 3MB messages seem to use least CPU in this run
• Overhead for large messages
• Otherwise quite independent of message frequency

Broker CPU usage 2

• 64 msg/s, 128 msg/s, 256 msg/s, 512 msg/s
• About 3MB messages seem to use least CPU in this run
• Overhead for large messages
• Otherwise quite independent of message frequency

Remaining CPU usage

• 20MB/s, 30MB/s, 40MB/s
• CPU usage of peer: EPICS, forwarder, verifier.
• This case, within statistics, seems linear CPU increase

Conclusions

• File writer design and structure
• Modular design, first round trip tests
• Finds data to be written in topics
• Wire format: identifier, timestamp, sourcename
• Streaming performance latency tests
• Latency is under control
• Non-negligible CPU usage on broker side
• Depends on bandwidth, not much on frequency