• Reference >
• MongoDB Package Components >
• mongoreplay

mongoreplay

Synopsis

New in version 3.4.

Availability

Available for Linux and macOS.

mongoreplay is a traffic capture and replay tool for MongoDB that you can use to inspect and record commands sent to a MongoDB instance, and then replay those commands back onto another host at a later time.

mongoreplay can help you preview how your MongoDB deployment will perform a production workload under a different environment, such as with a different storage engine, on different hardware, or with a different operating system configuration. mongoreplay can also help reproduce and investigate an issue by recording and replaying the operations that trigger the issue. Finally, mongoreplay serves as a more flexible version of the legacy mongosniff tool to help you investigate database activity.

Run mongoreplay from the system command line, not the mongo shell.

Required Access

mongoreplay requires access to the network interface that the record or monitor commands will listen on. You may need to run mongoreplay with root privileges to access the network device.

mongoreplay will not work with MongoDB instances using an SSL connection.

Warning

Only use root privileges when connecting to trusted sources.

If you are using play to connect to a MongoDB deployment that enforces access control, you must connect as a user with the required privileges to execute the recorded operations. Include the user’s credentials in the --host MongoDB connection string.

Options

mongoreplay

--verbosity, -v

Increases the amount of internal reporting returned on standard output or in log files. Increase the verbosity with the -v form by including the option multiple times, (e.g. -vvvvv.)

--debug, -d

Increases the amount of detail about mongoreplay operations and errors recorded in log files. Increase the debugging detail with the -d form by including the option multiple times, (e.g. -ddd.)

--silent, -s

When set, mongoreplay does not produce any log output.

--help

Returns information on the options and use of mongoreplay.

Commands

mongoreplay includes the following commands to record, play back, and monitor MongoDB network traffic.

record

record produces a playback file based on network traffic. record supports collecting network traffic directly or can accept a pcap file to produce the playback file. The playback file contains a list of all requests sent to the mongod instance during the recording as well as all responses transmitted to the client during the recording. The playback file also records metadata for each request, such as the connection identifier and timestamp.

The following prototype uses mongoreplay to record data on the loopback network interface and creates a playback file located at ~/recordings/playback.

copy

mongoreplay record -i eth0 -e "port 27017" -p ~/recordings/playback

Similarly, the following prototype uses mongoreplay to produce a playback file from an existing pcap file:

copy

mongoreplay record -f traffic.pcap -p ~/recordings/playback

record supports the following options:

mongoreplay record

record

-f <path>

Specifies the path to a pcap file that record should read to produce a playback file.

Use -f as an alternative to capturing network traffic using -i. You must specify either -f or -i. If you include both options, mongoreplay record produces an error.

-b <number>

Size of heap used to merge separate streams together.

--expr <filter expression>, -e <filter expression>

An expression in Berkeley Packet Filter (BPF) syntax to apply to incoming traffic to record. Required if you are capturing traffic from a network interface using -i.

For example, to capture traffic from a MongoDB instance running on port 27017, you would specify -e 'port 27017'.

-i <interface>

Specifies the network interface that record should listen on to capture network traffic.

Use with -e.

Use -i as an alternative to reading an existing pcap file with -i. You must specify either -f or -i. If you include both options, mongoreplay record produces an error.

--gzip <boolean>

If specified, record compresses the playback file with gzip.

--playback-file <path>, -p <path>

Specifies the path to which to write the playback file.

The produced playback file is a BSON file.

See

Use record for examples of using mongoreplay with the record command.

play

play replays a playback file created with record against a mongod instance.

The following prototype uses mongoreplay to replay the ~/recordings/playback playback file to the mongod instance running on 192.168.0.4:27018:

copy

mongoreplay play -p ~/recordings/playback --report ~/reports/replay_stats.json --host mongodb://192.168.0.4:27018

play supports the following options:

mongoreplay play

play

--collect <json|format|none>

Default: format

Specifies the output format for the collected statistics.

• json: outputs stat information as json
• format: uses the formatting specified in the --format option to produce the output file.
• none: does not provide any output

--report <path>

Specifies the path to which to write an execution report. Use --collect to specify the output format for the report.

If you do not specify --report, play writes to STDOUT.

--no-truncate

If specified, disables truncation of large reply payload data in the play log output.

--format

Default: %F{blue}%t%f %F{cyan}(Connection: %o:%i)%f %F{yellow}%l%f %F{red}%T %c%f %F{white}%n%f %F{green}%Q{Request:}%f%q%F{green}%R{Response:}%f%r)

Specifies the format for terminal output. You can specify arguments immediately after the format ‘verbs’ by wrapping them in curly braces, as in %Q{<arg>}.

If you specify --format, also specify format as the value for the --collect option.

--format supports the following verbs:

• %n: namespace
• %l: latency
• %t: time. You may optionally specify the date layout using the Go Programming Language’s time formatting. Go uses Mon Jan 2 15:04:05 MST 2006 as its reference time. You must specify the time format using the reference time. Thus, if you wanted to print the date in format yyyy-mm-dd hh:mm, you would specify %t{2006-01-02 15:04}. Refer to the Go time formatting documentation for more information.
• %T: op time
• %c: command
• %o: number of connections
• %i: request ID
• %q: request. You may optinally specified a dot-delimited field within the JSON structure, as in, %q{command_args.documents}.
• %r: response. You may optinally specified a dot-delimited field within the JSON structure, as in, %q{command_args.documents}.
• %Q{<arg>}: display <arg> on presence of request data
• %R{<arg>}: display <arg> on presence of response data

In addition, --format supports the following start/end ANSI escape sequences:

• %B/%b: bold
• %U/%u: underline
• %S/%s: standout
• %F/%f: text color (required arg – word or number, 8-color)
• %K/%k: background color (required arg – same as %F/%f)

--no-colors

When set, removes colors from the default format.

--playback-file <path>, -p <path>

Specifies the path from which to read the playback file.

If the playback file was created using the --gzip option, you must also specify --gzip when running play.

--speed number

Default: 1.0

Specifies a multiplier to adjust playback speed. --speed 1.0 processes the playback file in real time; --speed 0.5 at half speed; --speed 3.0 at triple speed.

The specified speed is a target speed. If mongoreplay play encounters a bottleneck, playback may be slower than the specified multiplier.

--host <hostname><:port>, -h <hostname><:port>

Default: localhost:27017

Specifies a MongoDB connection string for the MongoDB deployment to which to play back the captured network traffic.

By default, play attempts to connect to a mongod instance running on the localhost on port number 27017.

--repeat number

Default: 1

Specifies the number of times to play the playback file.

--queueTime number

Default: 15

Specifies the maximum time, in seconds, to queue operations in advance of transmitting them.

--no-preprocess

When set, play does not preprocess the input file to pre-map data such as MongoDB cursor IDs.

--gzip <boolean>

If specified, play decompresses the playback file with gzip.

See

Use play for examples of using mongoreplay with the play command.

monitor

monitor inspects live or pre-recorded MongoDB network traffic.

The following prototype uses mongoreplay to produce a JSON report based on the playback.bson playback file in the ~/recordings directory:

copy

mongoreplay monitor --collect json --report ~/reports/monitor-report.json -p ~/recordings/playback.bson

monitor supports the following options:

mongoreplay monitor

monitor

--collect <json|format|none>

Default: format

Specifies the output format for the collected statistics.

• json: outputs stat information as json
• format: uses the formatting specified in the --format option to produce the output file.
• none: does not provide any output

--report <path>

Specifies the path to which to write an execution report. Use --collect to specify the output format for the report.

If you do not specify --report, monitor writes to STDOUT.

--no-truncate

If specified, disables truncation of large reply payload data in the monitor log output.

--format

Default: %F{blue}%t%f %F{cyan}(Connection: %o:%i)%f %F{yellow}%l%f %F{red}%T %c%f %F{white}%n%f %F{green}%Q{Request:}%f%q%F{green}%R{Response:}%f%r)

Specifies the format for terminal output. You can specify arguments immediately after the format ‘verbs’ by wrapping them in curly braces, as in %Q{<arg>}.

If you specify --format, also specify format as the value for the --collect option.

--format supports the following verbs:

• %n: namespace
• %l: latency
• %t: time. You may optionally specify the date layout using the Go Programming Language’s time formatting. Go uses Mon Jan 2 15:04:05 MST 2006 as its reference time. You must specify the time format using the reference time. Thus, if you wanted to print the date in format yyyy-mm-dd hh:mm, you would specify %t{2006-01-02 15:04}. Refer to the Go time formatting documentation for more information.
• %T: op time
• %c: command
• %o: number of connections
• %i: request ID
• %q: request. You may optinally specified a dot-delimited field within the JSON structure, as in, %q{command_args.documents}.
• %r: response. You may optinally specified a dot-delimited field within the JSON structure, as in, %q{command_args.documents}.
• %Q{<arg>}: display <arg> on presence of request data
• %R{<arg>}: display <arg> on presence of response data

In addition, --format supports the following start/end ANSI escape sequences:

• %B/%b: bold
• %U/%u: underline
• %S/%s: standout
• %F/%f: text color (required arg – word or number, 8-color)
• %K/%k: background color (required arg – same as %F/%f)

--no-colors

When set, removes colors from the default format.

-f <path>

Specifies the path to a pcap file that monitor should read to produce a playback file.

Use -f as an alternative to capturing network traffic using -i. You must specify either -f or -i. If you include both options, mongoreplay monitor produces an error.

-b <number>

Size of heap used to merge separate streams together.

--expr <filter expression>, -e <filter expression>

An expression in Berkeley Packet Filter (BPF) syntax to apply to incoming traffic to record. Required if you are capturing traffic from a network interface using -i.

For example, to capture traffic from a MongoDB instance running on port 27017, you would specify -e 'port 27017'.

-i <interface>

Specifies the network interface that monitor should listen on to capture network traffic.

Use with -e.

Use -i as an alternative to reading an existing pcap file with -i. You must specify either -f or -i. If you include both options, mongoreplay monitor produces an error.

--paired

When specified, monitor outputs one line for each request/reply pair record.

--gzip <boolean>

If specified, monitor decompresses the playback file with gzip.

--playback-file <path>, -p <path>

Specifies the path from which to read the playback file.

See

Use monitor for examples of using mongoreplay with the monitor command.

mongoreplay Report Format

monitor and play can produce reports based on a playback file when run with the --report option.

Sample Record

The following is an example record from a JSON-formatted monitor report:

copy

{
   "order": 57,
   "op": "op_command",
   "command": "aggregate",
   "ns": "example",
   "request_data": {
      "command_args": {
         "aggregate": "restaurants",
         "cursor": {},
         "pipeline": [{
            "$match": {
               "borough": "Queens"
            }
         }, {
            "$group": {
               "_id": "$cuisine",
               "count": {
                  "$sum": 1.0
               }
            }
         }]
      },
      "input_docs": [],
      "metadata": {}
   },
   "reply_data": null,
   "connection_num": 0,
   "seen": "2016-09-08T10:21:48.072329-04:00",
   "request_id": 53
}

Fields

mongoreplay reports can include the following fields:

order

A monotonically increasing key indicating the order in which the operations were recorded and played back. This can be used to reconstruct the ordering of the series of operations executed on a connection, since the order in which they appear in the report file may not match the playback order.

op

The type of operation represented by the request: i.e. “query”, “insert”, “command”, “getmore”.

command

The name of the database command performed, such as isMaster or getLastError. This field is left blank for operations that are not commands, such as queries and inserts.

ns

The namespace on which the request was executed.

request_data

The payload of the operation.

• Query operations: request_data contains the actual query that was issued.
• Insert operations: request_data contains the documents being inserted.
• Update operations: request_data contains the query selector and the update modifier.

reply_data

The payload of the reply to the request.

nreturned

The number of documents returned as a result of the operation.

played_at

The time at which the play command executed the operation.

play_at

The time at which the operation was supposed to be executed by the play command.

playbacklag_us

The difference in microseconds in time between played_at and play_at. Higher values generally indicate that the target server is not able to keep up with the rate at which requests need to be executed according to the playback file.

connection_num

A key that identifies the connection on which the request was executed. All requests/replies that executed on the same connection have the same value for connection_num.

The connection_num value does not match the connection ID logged on the server side.

latency_us

The time difference in microseconds between when the request was sent by the client and when a response from the server was received.

errors

Lists any errors returned from the server.

msg

Lists the error message returned from the server.

seen

The time at which the operation was originally captured.

request_id

The ID of the MongoDB operation. The request_id for a request operation is the same as the response_id for the corresponding reply.

Output Formatting with --format

monitor and play output to either the terminal or, when run with --report (i.e. monitor --report or play --report), to a file.

Use the --collect (monitor --collect or play --collect) option to specify the format of the output:

• --collect json produces JSON output,
• --collect format outputs the data based on the formatting specified by the --format option for monitor and play .

Examples

Use record

Capture TCP data with record

To create a recording of traffic, use the record command. The following operation records traffic through port 27017 on the eth0 network interface and writes the captured traffic to ~/recordings/recording.bson:

copy

mongoreplay record -i eth0 -e "port 27017" -p ~/recordings/recording.bson

The produced playback file contains everything needed to re-execute the workload on another system.

Record a Playback File from Existing pcap Data

If you do not wish to use mongoreplay to capture traffic, you can capture traffic using a utility such as tcpdump and then create a mongoreplay recording from the static pcap file.

Warning

Only use root privileges when connecting to a trusted source.

The following operation uses tcpdump to capture traffic through port 27017 on the eth0 network interface and writes the captured data to a pcap file called traffic.pcap:

copy

sudo tcpdump -B $((100*1024)) -i eth0 -n "port 27017" -w traffic.pcap

To create the mongoreplay playback file, you can use record with the -f option to specify the pcap file from which to create the playback file, as in the following:

copy

mongoreplay record -f traffic.pcap -p ~/recordings/playback.bson

The produced playback file contains everything needed to re-execute the workload on another system.

Use play

Execute a Playback File Against a Target Host

play takes a playback file and executes the recorded operations against the mongodb://example.com:27018 host. Since the mongod enforces authentication, the connection string specified to --host also includes the username, password, and authentication database so that mongoreplay can write to the database.

copy

mongoreplay play -p ~/recordings/recording.bson --host mongodb://username:password@example.com:27018/admin

By default, ~bin.play executes the playback file at the rate of the recording. --speed lets you modify the playback speed. For example, the following operation executes the operations in recording.bson at twice the recording speed:

copy

mongoreplay play -p ~/recordings/recording.bson --speed=2.0 --host mongodb://username:password@example.com:27018/admin

Log Metrics About Execution Performance during Playback

play can produce a report with detailed metrics about the performance of each operation executed during playback.

The following example executes playback against the mongodb://example.com:27017 host and produces a JSON report written to ~/reports/playback-report.json

copy

mongoreplay play -p ~/recordings/recording.bson --report ~/reports/playback-report.json --collect json --host mongodb://example.com:27017

The play report contents would resemble:

copy

{
   "order": 2,
   "op": "op_command",
   "command": "listCollections",
   "ns": "dbs",
   "request_data": {
      "command_args": {
         "filter": {},
         "listCollections": 1.0
      },
      "input_docs": [],
      "metadata": {}
   },
   "reply_data": {
      "command_reply": {
         "cursor": {
            "firstBatch": [],
            "id": {
               "$numberLong": "0"
            },
            "ns": "dbs.$cmd.listCollections"
         },
         "ok": 1.0
      },
      "metadata": {},
      "output_docs": null
   },
   "played_at": "2016-09-12T15:50:47.576511488-04:00",
   "play_at": "2016-09-12T15:50:47.568587106-04:00",
   "playbacklag_us": 7924,
   "connection_num": 1,
   "latency_us": 327,
   "seen": "2016-08-29T15:18:46.000445Z",
   "request_id": 20
}
{
   "order": 4,
   "op": "op_command",
   "command": "aggregate",
   "ns": "example",
   "request_data": {
      "command_args": {
         "aggregate": "restaurants",
         "cursor": {},
         "pipeline": [{
            "$match": {
               "borough": "Manhattan"
            }
         }, {
            "$group": {
               "_id": "$cuisine"
            }
         }]
      },
      "input_docs": [],
      "metadata": {}
   },
   "reply_data": {
      "command_reply": {
         "cursor": {
            "firstBatch": [],
            "id": {
               "$numberLong": "0"
            },
            "ns": "example.restaurants"
         },
         "ok": 1.0
      },
      "metadata": {},
      "output_docs": null
   },
   "played_at": "2016-09-12T15:51:09.775716-04:00",
   "play_at": "2016-09-12T15:51:09.775244206-04:00",
   "playbacklag_us": 471,
   "connection_num": 1,
   "latency_us": 407,
   "seen": "2016-08-29T15:22:28.067016Z",
   "request_id": 21
}

Refer to mongoreplay Report Format for a description of each field.

Use monitor

Inspect Recorded Operations

monitor can create a report based on the contents of a playback file. monitor’s report includes all operations and some metadata about each operation’s execution.

The following operation uses monitor to create a JSON report based on the contents of the recording.bson playback file located in the ~/recordings directory and write the report to ~/reports/monitoring-report.json:

copy

mongoreplay monitor -p ~/recordings/recording.bson --report ~/reports/monitoring-report.json --collect json

The report contents would resemble:

copy

{
   "order": 0,
   "op": "command",
   "command": "isMaster",
   "ns": "admin.$cmd",
   "request_data": {
      "isMaster": 1
   },
   "reply_data": null,
   "connection_num": 0,
   "seen": "2016-08-26T14:45:59.387837Z",
   "request_id": 14
}
{
   "order": 1,
   "op": "reply",
   "request_data": null,
   "reply_data": {
      "ismaster": true,
      "localTime": {
         "$date": "2016-08-26T14:45:59.388Z"
      },
      "maxBsonObjectSize": 16777216,
      "maxMessageSizeBytes": 48000000,
      "maxWireVersion": 4,
      "maxWriteBatchSize": 1000,
      "minWireVersion": 0,
      "ok": 1.0
   },
   "nreturned": 1,
   "connection_num": 0,
   "latency_us": 214,
   "seen": "2016-08-26T14:45:59.388051Z",
   "request_id": 14
}
{
   "order": 2,
   "op": "op_command",
   "command": "listCollections",
   "ns": "folkore",
   "request_data": {
      "command_args": {
         "filter": {},
         "listCollections": 1.0
      },
      "input_docs": [],
      "metadata": {}
   },
   "reply_data": null,
   "connection_num": 0,
   "seen": "2016-08-26T14:45:59.388186Z",
   "request_id": 15
}

Refer to mongoreplay Report Format for a description of each field.

Inspect Live MongoDB Traffic

monitor can also inspect live traffic and, optionally, create a report based on the observed operations.

To monitor traffic in real time in your terminal, omit the --report option, as in the following:

copy

mongoreplay monitor -i eth0 -e 'port 27017' --collect json

Optionally, you can create a report based on the operations observed using monitor. The following example creates a JSON report written to ~/reports/monitor-live.json based on the traffic through port 27017 on the eth0 network interface:

copy

mongoreplay monitor -i eth0 -e 'port 27017' --report ~/reports/monitor-live.json --collect json

←   mongoperf mongoldap  →

© MongoDB, Inc 2008-present. MongoDB, Mongo, and the leaf logo are registered trademarks of MongoDB, Inc.