Commit Graph

546 Commits

Author SHA1 Message Date
Jon Chambers 3bf0188e7f Turn off alphanumeric sender ID for all countries. 2021-01-08 06:18:53 -05:00
Jon Chambers 91fc0fd623 Revert "Delete data in the storage service when deleting accounts."
This reverts commit ff1a721d5b.
2021-01-08 06:18:39 -05:00
Jon Chambers d2fcf68381 Record the status message when clients reject websocket messages. 2020-12-23 12:29:15 -05:00
Jon Chambers a4d0c17efd Record OS versions for iOS requests. 2020-12-23 11:36:31 -05:00
Jon Chambers ff1a721d5b Delete data in the storage service when deleting accounts. 2020-12-23 11:35:38 -05:00
Jon Chambers c870a1bbd5 Introduce a storage service client. 2020-12-23 11:35:38 -05:00
Ehren Kret ebf332a8c9
Record delivery duration excluding noise from non-primary devices (#311)
* Record delivery duration excluding noise from non-primary devices

* Extract method
2020-12-21 10:28:39 -06:00
Jon Chambers 85d1fff18f Actually increment the Android request counter. 2020-12-11 11:46:07 -05:00
Jon Chambers 6bb106c2cb Drop the Redis command timeout back down to 3 seconds to facilitate debug data collection. 2020-12-11 11:20:10 -05:00
Jon Chambers e551fd2c1b Revert "Pause checks for GV1 migration when checking for capability downgrades."
This reverts commit e7745db36e.
2020-12-10 17:02:41 -05:00
Jon Chambers 34a11c2338 Record OS versions for desktop and SDK versions for Android. 2020-12-10 17:02:05 -05:00
Jon Chambers 0de3a400eb Record unsuccessful server-to-client requests in more detail. 2020-12-10 17:01:46 -05:00
Jon Chambers e524ff965d Add a utility method for getting client platform tags from UA strings for metrics. 2020-12-10 17:01:46 -05:00
Jon Chambers 7ba689aaeb Measure adoption of the `gv1-migration` capability. 2020-12-09 19:08:52 -05:00
Jon Chambers 92fde83b3a Discard oversized messages bound for desktop clients via websockets. 2020-12-07 15:03:35 -05:00
Jon Chambers 3a268aef50 Reduce logging level for Lettuce connection events. 2020-12-07 11:56:41 -05:00
Jon Chambers f673bd8d7b Set device capabilities when linking a new device. 2020-12-02 13:21:08 -05:00
Ehren Kret 299b680013
Always include UUID in UD certificate (#300) 2020-12-01 08:56:55 -06:00
Jon Chambers 81e8352391 Time (and count) SQS "send message" operations. 2020-11-25 15:05:05 -05:00
Jon Chambers 1a627d6a87 Extend Redis command timeout to 3.5 seconds to avoid TCP retransmission "coincidences." 2020-11-25 15:04:06 -05:00
Ehren Kret 00a3e562dc
Force use of UCS-2 instead of GSM-7 for SMS to China (#297) 2020-11-20 14:41:48 -06:00
Jon Chambers 0628c9161c Use named threads for the JsonMetricsReporter executor service. 2020-11-18 15:46:14 -05:00
Jon Chambers 9b28672e19 Honor disabled metric attributes in JsonMetricsReporter. 2020-11-18 15:46:14 -05:00
Jon Chambers d764058a04 Measure contact intersection rate directly. 2020-11-18 14:28:53 -05:00
Jon Chambers 0aafe38496 Stop recording Lettuce latency metrics. 2020-11-17 13:20:37 -05:00
Jon Chambers e7745db36e Pause checks for GV1 migration when checking for capability downgrades. 2020-11-17 09:25:12 -05:00
Jon Chambers 474b879b16 Only notify CDS if an account attribute change actually changes an account's discoverability. 2020-11-16 10:54:12 -05:00
Jon Chambers 0a23b57ff8 Report Dropwizard metrics via the Wavefront proxy. 2020-11-13 17:14:13 -05:00
Jon Chambers 251e1b51c5 Make Micrometer batch size configurable. 2020-11-13 17:13:39 -05:00
Jon Chambers 2cc6c959a5 Revert "Temporarily suspend reporting of Lettuce latency metrics."
This reverts commit 2045153495a823b06334e7cbd86fb89c946c1cea.
2020-11-11 13:05:49 -05:00
Jon Chambers fb9aa672c9 Include the name of the calling thread when a command times out. 2020-11-11 13:05:35 -05:00
Jon Chambers 325e65db7f Expand UA parsing tests to cover OS details in desktop strings. 2020-11-11 13:05:18 -05:00
Jon Chambers 103b49ec45 Record the number of non-success responses from clients when sending messages via websockets. 2020-11-10 11:47:57 -05:00
Jon Chambers 6c78d7544f Capture a thread dump when Redis commands time out. 2020-11-10 11:47:39 -05:00
Jon Chambers 7cf50a15d0 Include client age/UA string when closing due to a spurious keepalive request. 2020-11-10 11:45:12 -05:00
Jon Chambers adbc4e9fec Record the platforms of clients that send a keepalive without a local presence. 2020-11-10 11:45:12 -05:00
Jon Chambers 4815434dd7 Record the platforms of clients that are getting displaced. 2020-11-10 11:45:12 -05:00
Jon Chambers b25e50bdae Drop API keys from Micrometer configuration. 2020-11-09 09:26:56 -05:00
Ehren Kret 604287244f Update copyright statement on all source files
IntelliJ Copyright Profile used to automate this.
2020-11-04 11:55:35 -05:00
Jon Chambers 4a4a721e90 Log timeouts in addition to incrementing a counter to make it easier to get precise timestamps. 2020-10-30 11:35:59 -04:00
Jon Chambers a4062b338e Count timeouts directly. 2020-10-29 10:51:18 -04:00
Ehren Kret 5587b7d469 Expose gv1-migration on profile endpoint 2020-10-28 13:00:57 -04:00
Ehren Kret 26870d134f Set source UUID when delivering envelopes from message cache/db on websocket 2020-10-28 12:38:32 -04:00
Jon Chambers 0431a2abb1 De-dupe connection event logging messages. 2020-10-28 12:29:14 -04:00
Ehren Kret c2db2d3cbd
Add GV1 Migration capability 2020-10-27 16:17:21 -04:00
Jon Chambers 05d9ec673e
Send push notifications if websockets close before all messages are delivered 2020-10-27 16:02:55 -04:00
Jon Chambers 1732cf9243 Add filters/tasks to enable/disable request logging. 2020-10-23 11:35:06 -04:00
Jon Chambers ab62c19de9 Temporarily suspend reporting of Lettuce latency metrics. 2020-10-23 11:30:42 -04:00
Jon Chambers 96d3a69479 Use container-managed executors for APN/GCM senders. 2020-10-23 11:30:03 -04:00
Jon Chambers 8523bb1ad8 Change the "oversized message" threshold from 64kB to 1MB. 2020-10-23 11:13:19 -04:00
Jon Chambers e6da54d9b8 Resolve build error introduced while merging. 2020-10-20 19:04:44 -04:00
Jon Chambers 0a843dc086 Tighten the "prune peers" interval; move from fixed-rate to fixed-delay scheduling. 2020-10-20 19:00:55 -04:00
Jon Chambers 7b3ed2dcbf Catch exceptions thrown while pruning missing peers. 2020-10-20 19:00:55 -04:00
Jon Chambers 42ed6c3ded Add clients to the "cleanup" list before actually setting their presence keys. 2020-10-20 19:00:55 -04:00
Jon Chambers 23ca011ac1 Record account deletion reasons. 2020-10-20 19:00:34 -04:00
Jon Chambers d82b3dc429 Record a count of deleted accounts by country. 2020-10-20 19:00:34 -04:00
Jon Chambers e391793c58 Remove now-redundant Redis execution time metrics. 2020-10-20 19:00:11 -04:00
Jon Chambers 236cef4b56 Report Lettuce command latency via Micrometer. 2020-10-20 19:00:11 -04:00
Jon Chambers 45687513bf Revert "Revert "Share resources between Lettuce clients.""
This reverts commit 334f509be599fa6a501026e900d912ff7187e150.
2020-10-20 19:00:11 -04:00
Jon Chambers 019ffdaf12 Add a command for dumping Redis command stats. 2020-10-20 18:59:44 -04:00
Jon Chambers df847431eb Measure total bytes written to websockets and failed send attempts. 2020-10-20 17:22:30 -04:00
Jon Chambers 99f488d48f Drop websocket connection names (unused for a while now). 2020-10-19 11:24:35 -04:00
Jon Chambers 05929871c9 Rename PushSender to MessageSender and add docs. 2020-10-19 11:24:35 -04:00
Jon Chambers 74b3daa70a Collapse WebsocketSender into PushSender. 2020-10-19 11:24:35 -04:00
Jon Chambers 5e30b0499a Move provisioning message-sending to its own manager class. 2020-10-19 11:24:35 -04:00
Jon Chambers 85c7347899 Add a command for dumping Redis SLOWLOG output. 2020-10-15 12:18:37 -04:00
Jon Chambers 3a84775912 Log cluster topology change events, too. 2020-10-13 16:07:08 -04:00
Jon Chambers 290a82e61c Log when Lettuce connection events happen. 2020-10-13 16:07:08 -04:00
Jon Chambers adac7d7fb2 Estimate the size of message entity lists sent via the REST API. 2020-10-13 15:49:11 -04:00
Jon Chambers 52320ebb91 Revert "Share resources between Lettuce clients."
This reverts commit eab1f503a5.
2020-10-13 12:44:54 -04:00
Jon Chambers eab1f503a5 Share resources between Lettuce clients. 2020-10-11 14:36:28 -04:00
Jon Chambers a9d0aa136d Add OS-reported metrics for cached/buffered memory. 2020-10-11 13:43:15 -04:00
Jon Chambers 691ab3080d Fix some metrics names/types. 2020-10-11 12:37:17 -04:00
Jon Chambers c5147e0c68 Report direct memory metrics. 2020-10-11 11:37:51 -04:00
Jon Chambers e9b0829860 Report the maximum number of file descriptors allowed by the OS. 2020-10-11 11:27:57 -04:00
Jon Chambers 95428ab8b0 Report GC metrics. 2020-10-11 11:08:24 -04:00
Jon Chambers 775d56fe52 Drop the "repair message queue metadata" script. 2020-10-09 18:18:30 -04:00
Jon Chambers ac2ff29288 Make sure to close scheduled reporters. 2020-10-09 18:05:00 -04:00
Jon Chambers 8e1975efe4 Record the number of deletable accounts per crawled chunk. 2020-10-08 10:51:41 -04:00
Curt Brune 39c09733d3
Add /v1/payments/auth endpoint 2020-10-08 10:51:01 -04:00
Jon Chambers e1c397993d
Require Android clients to support the gv2-3 capability 2020-10-06 16:49:49 -04:00
Jon Chambers 58ca4baf71 Time account deletion operations. 2020-10-06 11:04:47 -04:00
Jon Chambers 5245b68689 Remove temporary metrics. 2020-10-06 11:04:47 -04:00
Jon Chambers 2b6811cb1b Really delete old accounts instead of just removing their push channels. 2020-10-06 11:04:47 -04:00
Jon Chambers c82496b972 Remove the "repair queue metadata" script. 2020-10-05 16:57:16 -04:00
Jon Chambers c31348ea9a Drop the "insert messages" timeout. 2020-10-05 16:57:01 -04:00
Jon Chambers c885540749 Check that the return of ZRANGEBYSCORE isn't an empty list. 2020-10-05 10:38:40 -04:00
Jon Chambers bb087caddc Don't panic if a queue exists, but is empty when repairing metadata. 2020-10-04 16:09:56 -04:00
Jon Chambers 5e3f8b9c2e Disallow insertion of duplicate messages. 2020-10-04 15:34:14 -04:00
Jon Chambers 1ccfe928f7 Add a test to make sure that we don't double-insert messages with the same GUID. 2020-10-04 15:34:14 -04:00
Jon Chambers 3016269268 Revert "Temporarily disable the message persisters entirely."
This reverts commit d464721397.
2020-10-04 15:25:06 -04:00
Jon Chambers 952cfae4e6 Repair queue metadata before persisting queues. 2020-10-04 15:25:06 -04:00
Jon Chambers df7f209ebc Revert "Don't insert message batches in transactions."
This reverts commit 16eefe333f.
2020-10-04 15:12:15 -04:00
Jon Chambers d464721397 Temporarily disable the message persisters entirely. 2020-10-04 11:44:35 -04:00
Jon Chambers 551a85c1e6 Use named variables instead of referring to KEYS/ARGV array indices in message cache scripts. 2020-10-04 11:27:27 -04:00
Jon Chambers 2686761608 Instrument "get queues to persist" calls and "persist queues" exceptions. 2020-10-04 10:48:42 -04:00
Jon Chambers 02a2c3224f Discard unused feature flag constants/mocking. 2020-10-04 10:48:42 -04:00
Jon Chambers 8ec1dda9ba Give the persister worker thread a meaningful name. 2020-10-04 10:48:42 -04:00
Jon Chambers 0308532523 Set a query timeout of 5 seconds when inserting batches of messages. 2020-10-04 10:48:42 -04:00
Jon Chambers 10b3af2947 Revert "Insert messages individually."
This reverts commit 158bfe4816.
2020-10-04 10:48:42 -04:00
Jon Chambers 158bfe4816 Insert messages individually. 2020-10-03 13:13:34 -04:00
Jon Chambers 16eefe333f Don't insert message batches in transactions. 2020-10-03 11:43:42 -04:00
Jon Chambers 65e585e122 Pause only if we're running low on queues to persist. 2020-10-03 11:43:34 -04:00
Jon Chambers 2ba36ee04c Add a gauge for worker thread liveness. 2020-10-03 11:43:34 -04:00
Jon Chambers fc05529574 Let MessagePersister manage its own worker thread. 2020-10-03 11:43:34 -04:00
Jon Chambers 07d24f487a Don't re-register metrics for shared circuit breakers. 2020-10-02 15:05:00 -04:00
Jon Chambers 811acdb7f5 Use separate namespaces for Redis breaker/retry metrics. 2020-10-02 10:57:05 -04:00
Jon Chambers a7266364d1 Refactor peer pruning to be more retry-friendly. 2020-10-01 17:17:07 -04:00
Jon Chambers e83b41dc01 Reduce default Redis cluster command timeout to 3 seconds. 2020-10-01 17:17:07 -04:00
Jon Chambers 76665dd56e Retry Redis commands that time out. 2020-10-01 17:17:07 -04:00
Jon Chambers 2d42b478ba Consolidate cluster and pub/sub circuit breakers. 2020-10-01 17:17:07 -04:00
Jon Chambers 885fa6beae Add tests for Device#isEnabled. 2020-10-01 12:54:35 -04:00
Jon Chambers 65cdd5fcbe Drop the 365-day check when deciding if an account is enabled. 2020-10-01 12:54:35 -04:00
Jon Chambers 4302e19aba Register a UUID argument factory for the messages database. 2020-10-01 11:06:43 -04:00
Jon Chambers 0c6f05f34a Add a (failing!) test for sending a sealed-sender message after a non-sealed-sender message. 2020-10-01 11:06:43 -04:00
Jon Chambers 8040c285cd Include stack traces when reporting persistence issues. 2020-09-30 11:47:16 -04:00
Jon Chambers ada454f56f Add a meter for persisting individual messages. 2020-09-30 10:39:56 -04:00
Jon Chambers 57d2ef8740 Return queues to the "to persist" list if something goes wrong during persistence. 2020-09-30 10:39:56 -04:00
Jon Chambers a97e0982e3 Add an integration test for message persistence. 2020-09-30 10:39:56 -04:00
Jon Chambers eaa2060d84 Fix an incorrect locking key and some previously-suppressed lock contention issues. 2020-09-30 10:39:56 -04:00
Jon Chambers 3e02c574e7 Log exceptions when persisting messages. 2020-09-30 10:39:56 -04:00
Jon Chambers c7230ccbb0 Remove messages from the cache in bulk. 2020-09-29 10:58:02 -04:00
Jon Chambers fc71ced660 Persist messages in batches. 2020-09-29 10:58:02 -04:00
Jon Chambers 6041a9d094 Make exit conditions slightly more conservative. 2020-09-29 10:58:02 -04:00
Jon Chambers 599cd766e1 Let Dropwizard manage persister thread lifecycles. 2020-09-29 10:58:02 -04:00
Alan Evans e64c8007c0
Detect GV2 capability in non-gcm Android devices 2020-09-28 15:54:10 -04:00
Jon Chambers 9339823e84 Add temporary metrics to monitor the ratio of enabled/disabled accounts. 2020-09-28 15:33:52 -04:00
Jon Chambers e6d4620af1 Only allow linking desktop clients if they support the third-generation GV2 capability. 2020-09-25 17:08:32 -04:00
Jon Chambers 656e6db846 Only consider desktop devices GV2-capable if they send the third-gen GV2 capability. 2020-09-25 17:08:32 -04:00
Jon Chambers 30474e3a2b Add a test for message ordering. 2020-09-25 11:41:58 -04:00
Jon Chambers 460bd98f1b Add metrics for messages missing GUIDs. 2020-09-25 11:41:22 -04:00
Jon Chambers a553eba574 Add an API endpoint for deleting accounts. 2020-09-25 11:39:17 -04:00
Jon Chambers 61f515670c Add plumbing for deleting accounts and all associated data. 2020-09-25 11:39:17 -04:00
Jon Chambers 789af0f8a6 Add support for deleting keys associated with an account. 2020-09-25 11:39:17 -04:00
Jon Chambers 86fae58c96 Add support for deleting account entities from the database. 2020-09-25 11:39:17 -04:00
Jon Chambers c54d3abe47 Check for the second-gen GV2 capability when linking devices. 2020-09-24 19:04:02 -04:00
Jon Chambers 6fe511eb50 Fix a bad size check when loading stored messages. 2020-09-23 18:02:33 -04:00
Jon Chambers 17d18b22c7 Drop pub/sub sending logic from WebsocketSender. 2020-09-23 14:51:02 -04:00
Jon Chambers 66a04ed730 Don't explicitly notify clients when messages get persisted. 2020-09-23 14:51:02 -04:00
Jon Chambers 7e14a0bc30 Drop pub/sub operations from WebsocketConnection. 2020-09-23 14:51:02 -04:00
Jon Chambers 77de0f86dc Require desktop clients to send the new gv2-2 capability flag. 2020-09-23 12:05:58 -04:00
Jon Chambers 3b4bc9163a Untangle thread pool names, tweak sizes, and add instrumentation. 2020-09-22 10:21:33 -04:00
Jon Chambers e146135bd1 Don't attempt to send more messages if sending failed for any reason. 2020-09-22 10:21:33 -04:00
Jon Chambers e9e18afb4a Add a (failing) integration test demonstrating an infinite loop. 2020-09-22 10:21:33 -04:00
Jon Chambers 62c31eb202 Revert "Revert keyspace delivery for all messages"
This reverts commit 4dc49604b6.
2020-09-22 10:21:33 -04:00
Jon Chambers 1eacee85ae Count how many iOS users set the old GV2 capability flag. 2020-09-21 18:58:07 -04:00
Jon Chambers 5986145282 Add a second-generation GV2 capability and ignore the old capability for iOS devices. 2020-09-21 18:57:53 -04:00
Jon Chambers b134a69a28 Record the number of authentications for users with/without GV2 support. 2020-09-21 15:42:13 -04:00
Jon Chambers 83f9eacac4 Refactor UserAgentTagUtil to parse UA strings with UserAgentUtil. 2020-09-21 12:24:08 -04:00
Jon Chambers baab6b951b Add a general utility class for parsing user-agent strings. 2020-09-21 12:24:08 -04:00