408b959441
Require a push challenge when registering (or else require a captcha).
2021-02-01 20:44:21 -05:00
35fc98a188
Add an experiment enrollment manager.
2021-02-01 11:08:16 -05:00
92f6a79e1f
Add a dynamic configuration manager
2021-02-01 11:01:58 -05:00
8f94ed68a3
Ignore expired devices when checking for GV1->GV2 migration capability.
2021-01-30 16:55:05 -05:00
ce1a4b94cb
Actually store emoji/about text in the database.
2021-01-27 10:34:13 -05:00
92a0deffcf
Add more robust tests for about/emoji fields.
2021-01-27 10:34:13 -05:00
97b6f6028b
Fix a minor typo in the help text for a feature flag task.
2021-01-25 18:03:38 -05:00
611e8c39ee
Actually drop feature flag config.
2021-01-25 15:20:06 -05:00
01f1c263a6
Add a meter for captcha requests.
2021-01-25 14:58:27 -05:00
24ea6a9f1d
Revert "Temporarily disable registration abuse system"
...
This reverts commit 22ef058cb6
2021-01-25 14:58:27 -05:00
46c800b8b7
Smoosh request logging tasks together rather than having one task for each direction.
2021-01-25 14:58:15 -05:00
f10be893ce
Drop the old feature flag controller.
2021-01-25 14:55:57 -05:00
c606c1664f
Add admin tasks for listing, setting, and deleting feature flags.
2021-01-25 14:55:57 -05:00
225932b4c9
Add emojis/"about" text to profiles
2021-01-20 15:42:47 -05:00
6b850b9894
Allow (versioned) profile names up to 380 base64 characters long.
2021-01-20 11:08:10 -05:00
943a5d1036
Shard push scheduling cache
2021-01-19 15:50:12 -05:00
b25da8ceaa
Don't attempt SMS to iran ( #355 )
2021-01-19 09:13:37 -08:00
10cdb7387d
Be consistent with use of DataSize class
2021-01-18 17:01:43 -06:00
dd436dd1dd
Create a Meter for tracking messages larger than 256kib
2021-01-18 17:01:43 -06:00
13b84635b5
Drop an unused message database index. ( #352 )
2021-01-18 10:26:03 -06:00
27534d408f
Log when messages cache detects topology change ( #354 )
...
Co-authored-by: Moxie Marlinspike <moxie+github@signal.org>
2021-01-17 17:13:23 -08:00
0a23ce870a
Allow message persisters to be disabled by a feature flag.
2021-01-17 11:13:12 -05:00
c355ef8d53
Reduce the message cache thread pool size.
2021-01-16 11:15:25 -05:00
1feb23ba99
Stop periodic topology refreshes.
2021-01-16 03:35:36 -05:00
59a0fd0799
Embiggen message cluster thread pool.
2021-01-16 02:57:04 -05:00
00b5cfcf17
Allow the client presence manager to use an entirely separate cluster.
2021-01-16 02:57:04 -05:00
9e342f253d
Use the same client for inserts and reads in the message cache cluster.
2021-01-16 01:50:40 -05:00
20c48b6bb2
Expand message-related thread pools to 1 thread per shard.
2021-01-16 01:50:40 -05:00
4f9e7bb572
Separate Lettuce thread pools.
2021-01-16 01:18:05 -05:00
0a322d5a9f
Add a "doomsday switch."
2021-01-15 18:05:18 -05:00
59eb6d10c1
Gate based on destination rather than random.
2021-01-15 18:05:18 -05:00
a57ce1dd17
Add machinery to allow a percentage of message sends to succeed.
2021-01-15 17:05:16 -05:00
b100b3c36b
Reject traffic without logging exceptions
2021-01-15 16:23:53 -05:00
81c1ba6eef
Respond to all "message send" attempts with HTTP/503.
2021-01-15 15:34:14 -05:00
93ae4d1ee6
Move the client presence manager to its own breaker.
2021-01-15 13:51:39 -05:00
9c53d818f4
Use separate clusters for message cache read/write operations.
2021-01-15 13:51:39 -05:00
e5a2c1ab10
Always return an empty list of prekeys.
2021-01-15 12:27:10 -05:00
67ed035b36
Retry serializable key transactions.
2021-01-13 17:38:29 -05:00
ad30786f4a
Parallelize message persisters.
2021-01-12 18:50:14 -05:00
2e01da5ec1
Add a task to enable/disable accelerated crawling.
2021-01-11 19:29:18 -05:00
8fb37a0024
Log when a crawling cycle has wrapped up.
2021-01-11 19:29:18 -05:00
9412a7424c
Return HTTP/429 whenever somebody tries to get contacts from the old directory system.
2021-01-11 19:29:10 -05:00
f8cbb4f386
Temporarily suspend client version metrics to reduce load on our metric aggregator.
2021-01-11 14:04:44 -05:00
86ccaa52a5
Allow configuration of multiple directory account crawler listeners ( #325 )
...
* Allow configuration of multiple directory account crawler listeners
Only one should update the local redis directory. This one is marked
with replicationPrimary true. The others in the list only serve to
issue replication requests over to CDS replication load balancers.
* Update one more metric name
2021-01-10 17:11:02 -06:00
cc3e5d23e4
Enable Lettuce adaptive topology refreshes.
2021-01-10 16:20:35 -05:00
cac86d1f77
Standardize toplogy event handling strategy.
2021-01-10 15:14:12 -05:00
22f7bb822f
Raise log level of toplogy changes.
2021-01-10 15:14:12 -05:00
1b53f10091
Reload scripts across the whole cluster if one shard is missing the script.
2021-01-10 15:00:12 -05:00
bac268a21c
Don't send a reply to clients until messages are safely in a non-volatile store.
2021-01-10 13:03:40 -05:00
321e6e6679
Don't validate cluster membership (allow new shards to join dynamically).
2021-01-10 12:58:35 -05:00
22ef058cb6
Temporarily disable registration abuse system
2021-01-09 15:57:55 -05:00
9ee6419bc0
Publish directory updates to multiple SQS queues.
2021-01-08 18:07:18 -05:00
3bf0188e7f
Turn off alphanumeric sender ID for all countries.
2021-01-08 06:18:53 -05:00
91fc0fd623
Revert "Delete data in the storage service when deleting accounts."
...
This reverts commit ff1a721d5b
2021-01-08 06:18:39 -05:00
d2fcf68381
Record the status message when clients reject websocket messages.
2020-12-23 12:29:15 -05:00
a4d0c17efd
Record OS versions for iOS requests.
2020-12-23 11:36:31 -05:00
ff1a721d5b
Delete data in the storage service when deleting accounts.
2020-12-23 11:35:38 -05:00
c870a1bbd5
Introduce a storage service client.
2020-12-23 11:35:38 -05:00
ebf332a8c9
Record delivery duration excluding noise from non-primary devices ( #311 )
...
* Record delivery duration excluding noise from non-primary devices
* Extract method
2020-12-21 10:28:39 -06:00
85d1fff18f
Actually increment the Android request counter.
2020-12-11 11:46:07 -05:00
6bb106c2cb
Drop the Redis command timeout back down to 3 seconds to facilitate debug data collection.
2020-12-11 11:20:10 -05:00
e551fd2c1b
Revert "Pause checks for GV1 migration when checking for capability downgrades."
...
This reverts commit e7745db36e
2020-12-10 17:02:41 -05:00
34a11c2338
Record OS versions for desktop and SDK versions for Android.
2020-12-10 17:02:05 -05:00
0de3a400eb
Record unsuccessful server-to-client requests in more detail.
2020-12-10 17:01:46 -05:00
e524ff965d
Add a utility method for getting client platform tags from UA strings for metrics.
2020-12-10 17:01:46 -05:00
7ba689aaeb
Measure adoption of the `gv1-migration` capability.
2020-12-09 19:08:52 -05:00
92fde83b3a
Discard oversized messages bound for desktop clients via websockets.
2020-12-07 15:03:35 -05:00
3a268aef50
Reduce logging level for Lettuce connection events.
2020-12-07 11:56:41 -05:00
f673bd8d7b
Set device capabilities when linking a new device.
2020-12-02 13:21:08 -05:00
299b680013
Always include UUID in UD certificate ( #300 )
2020-12-01 08:56:55 -06:00
81e8352391
Time (and count) SQS "send message" operations.
2020-11-25 15:05:05 -05:00
1a627d6a87
Extend Redis command timeout to 3.5 seconds to avoid TCP retransmission "coincidences."
2020-11-25 15:04:06 -05:00
00a3e562dc
Force use of UCS-2 instead of GSM-7 for SMS to China ( #297 )
2020-11-20 14:41:48 -06:00
0628c9161c
Use named threads for the JsonMetricsReporter executor service.
2020-11-18 15:46:14 -05:00
9b28672e19
Honor disabled metric attributes in JsonMetricsReporter.
2020-11-18 15:46:14 -05:00
d764058a04
Measure contact intersection rate directly.
2020-11-18 14:28:53 -05:00
0aafe38496
Stop recording Lettuce latency metrics.
2020-11-17 13:20:37 -05:00
e7745db36e
Pause checks for GV1 migration when checking for capability downgrades.
2020-11-17 09:25:12 -05:00
474b879b16
Only notify CDS if an account attribute change actually changes an account's discoverability.
2020-11-16 10:54:12 -05:00
0a23b57ff8
Report Dropwizard metrics via the Wavefront proxy.
2020-11-13 17:14:13 -05:00
251e1b51c5
Make Micrometer batch size configurable.
2020-11-13 17:13:39 -05:00
217d270457
Update to Lettuce 6.0.1.
2020-11-13 10:50:21 -05:00
143b6f0df1
Revert "Add a debug version of Lettuce to track down the cause of https://github.com/lettuce-io/lettuce-core/issues/1494 ."
...
This reverts commit 4d5fbec5a5
2020-11-13 10:50:21 -05:00
2cc6c959a5
Revert "Temporarily suspend reporting of Lettuce latency metrics."
...
This reverts commit 2045153495a823b06334e7cbd86fb89c946c1cea.
2020-11-11 13:05:49 -05:00
fb9aa672c9
Include the name of the calling thread when a command times out.
2020-11-11 13:05:35 -05:00
325e65db7f
Expand UA parsing tests to cover OS details in desktop strings.
2020-11-11 13:05:18 -05:00
103b49ec45
Record the number of non-success responses from clients when sending messages via websockets.
2020-11-10 11:47:57 -05:00
6c78d7544f
Capture a thread dump when Redis commands time out.
2020-11-10 11:47:39 -05:00
4d5fbec5a5
Add a debug version of Lettuce to track down the cause of https://github.com/lettuce-io/lettuce-core/issues/1494 .
2020-11-10 11:45:46 -05:00
7cf50a15d0
Include client age/UA string when closing due to a spurious keepalive request.
2020-11-10 11:45:12 -05:00
adbc4e9fec
Record the platforms of clients that send a keepalive without a local presence.
2020-11-10 11:45:12 -05:00
4815434dd7
Record the platforms of clients that are getting displaced.
2020-11-10 11:45:12 -05:00
b25e50bdae
Drop API keys from Micrometer configuration.
2020-11-09 09:26:56 -05:00
604287244f
Update copyright statement on all source files
...
IntelliJ Copyright Profile used to automate this.
2020-11-04 11:55:35 -05:00
4a4a721e90
Log timeouts in addition to incrementing a counter to make it easier to get precise timestamps.
2020-10-30 11:35:59 -04:00
a4062b338e
Count timeouts directly.
2020-10-29 10:51:18 -04:00
5587b7d469
Expose gv1-migration on profile endpoint
2020-10-28 13:00:57 -04:00
26870d134f
Set source UUID when delivering envelopes from message cache/db on websocket
2020-10-28 12:38:32 -04:00
fb2baad7cc
Restore netty-tcnative.
2020-10-28 12:29:30 -04:00
0431a2abb1
De-dupe connection event logging messages.
2020-10-28 12:29:14 -04:00
c2db2d3cbd
Add GV1 Migration capability
2020-10-27 16:17:21 -04:00
05d9ec673e
Send push notifications if websockets close before all messages are delivered
2020-10-27 16:02:55 -04:00
1732cf9243
Add filters/tasks to enable/disable request logging.
2020-10-23 11:35:06 -04:00
ab62c19de9
Temporarily suspend reporting of Lettuce latency metrics.
2020-10-23 11:30:42 -04:00
96d3a69479
Use container-managed executors for APN/GCM senders.
2020-10-23 11:30:03 -04:00
8523bb1ad8
Change the "oversized message" threshold from 64kB to 1MB.
2020-10-23 11:13:19 -04:00
169c3d5a0f
Update to Pushy 0.14.2.
2020-10-21 15:20:36 -04:00
9cffbe3d49
Drop netty-tcnative-boringssl-static as a dependency.
2020-10-21 15:20:36 -04:00
e6da54d9b8
Resolve build error introduced while merging.
2020-10-20 19:04:44 -04:00
0a843dc086
Tighten the "prune peers" interval; move from fixed-rate to fixed-delay scheduling.
2020-10-20 19:00:55 -04:00
7b3ed2dcbf
Catch exceptions thrown while pruning missing peers.
2020-10-20 19:00:55 -04:00
42ed6c3ded
Add clients to the "cleanup" list before actually setting their presence keys.
2020-10-20 19:00:55 -04:00
23ca011ac1
Record account deletion reasons.
2020-10-20 19:00:34 -04:00
d82b3dc429
Record a count of deleted accounts by country.
2020-10-20 19:00:34 -04:00
e391793c58
Remove now-redundant Redis execution time metrics.
2020-10-20 19:00:11 -04:00
236cef4b56
Report Lettuce command latency via Micrometer.
2020-10-20 19:00:11 -04:00
45687513bf
Revert "Revert "Share resources between Lettuce clients.""
...
This reverts commit 334f509be599fa6a501026e900d912ff7187e150.
2020-10-20 19:00:11 -04:00
019ffdaf12
Add a command for dumping Redis command stats.
2020-10-20 18:59:44 -04:00
1a57d4fe11
Update to Lettuce 6.
2020-10-20 18:59:26 -04:00
df847431eb
Measure total bytes written to websockets and failed send attempts.
2020-10-20 17:22:30 -04:00
99f488d48f
Drop websocket connection names (unused for a while now).
2020-10-19 11:24:35 -04:00
05929871c9
Rename PushSender to MessageSender and add docs.
2020-10-19 11:24:35 -04:00
74b3daa70a
Collapse WebsocketSender into PushSender.
2020-10-19 11:24:35 -04:00
5e30b0499a
Move provisioning message-sending to its own manager class.
2020-10-19 11:24:35 -04:00
85c7347899
Add a command for dumping Redis SLOWLOG output.
2020-10-15 12:18:37 -04:00
3a84775912
Log cluster topology change events, too.
2020-10-13 16:07:08 -04:00
290a82e61c
Log when Lettuce connection events happen.
2020-10-13 16:07:08 -04:00
adac7d7fb2
Estimate the size of message entity lists sent via the REST API.
2020-10-13 15:49:11 -04:00
52320ebb91
Revert "Share resources between Lettuce clients."
...
This reverts commit eab1f503a5
2020-10-13 12:44:54 -04:00
eab1f503a5
Share resources between Lettuce clients.
2020-10-11 14:36:28 -04:00
a9d0aa136d
Add OS-reported metrics for cached/buffered memory.
2020-10-11 13:43:15 -04:00
691ab3080d
Fix some metrics names/types.
2020-10-11 12:37:17 -04:00
c5147e0c68
Report direct memory metrics.
2020-10-11 11:37:51 -04:00
e9b0829860
Report the maximum number of file descriptors allowed by the OS.
2020-10-11 11:27:57 -04:00
95428ab8b0
Report GC metrics.
2020-10-11 11:08:24 -04:00
775d56fe52
Drop the "repair message queue metadata" script.
2020-10-09 18:18:30 -04:00
ac2ff29288
Make sure to close scheduled reporters.
2020-10-09 18:05:00 -04:00
8e1975efe4
Record the number of deletable accounts per crawled chunk.
2020-10-08 10:51:41 -04:00
39c09733d3
Add /v1/payments/auth endpoint
2020-10-08 10:51:01 -04:00
e1c397993d
Require Android clients to support the gv2-3 capability
2020-10-06 16:49:49 -04:00
58ca4baf71
Time account deletion operations.
2020-10-06 11:04:47 -04:00
5245b68689
Remove temporary metrics.
2020-10-06 11:04:47 -04:00
2b6811cb1b
Really delete old accounts instead of just removing their push channels.
2020-10-06 11:04:47 -04:00
c82496b972
Remove the "repair queue metadata" script.
2020-10-05 16:57:16 -04:00
c31348ea9a
Drop the "insert messages" timeout.
2020-10-05 16:57:01 -04:00
c885540749
Check that the return of ZRANGEBYSCORE isn't an empty list.
2020-10-05 10:38:40 -04:00
bb087caddc
Don't panic if a queue exists, but is empty when repairing metadata.
2020-10-04 16:09:56 -04:00
5e3f8b9c2e
Disallow insertion of duplicate messages.
2020-10-04 15:34:14 -04:00
1ccfe928f7
Add a test to make sure that we don't double-insert messages with the same GUID.
2020-10-04 15:34:14 -04:00
3016269268
Revert "Temporarily disable the message persisters entirely."
...
This reverts commit d464721397
2020-10-04 15:25:06 -04:00
952cfae4e6
Repair queue metadata before persisting queues.
2020-10-04 15:25:06 -04:00
df7f209ebc
Revert "Don't insert message batches in transactions."
...
This reverts commit 16eefe333f
2020-10-04 15:12:15 -04:00
d464721397
Temporarily disable the message persisters entirely.
2020-10-04 11:44:35 -04:00
551a85c1e6
Use named variables instead of referring to KEYS/ARGV array indices in message cache scripts.
2020-10-04 11:27:27 -04:00
2686761608
Instrument "get queues to persist" calls and "persist queues" exceptions.
2020-10-04 10:48:42 -04:00
02a2c3224f
Discard unused feature flag constants/mocking.
2020-10-04 10:48:42 -04:00
8ec1dda9ba
Give the persister worker thread a meaningful name.
2020-10-04 10:48:42 -04:00
0308532523
Set a query timeout of 5 seconds when inserting batches of messages.
2020-10-04 10:48:42 -04:00
10b3af2947
Revert "Insert messages individually."
...
This reverts commit 158bfe4816
2020-10-04 10:48:42 -04:00
158bfe4816
Insert messages individually.
2020-10-03 13:13:34 -04:00
16eefe333f
Don't insert message batches in transactions.
2020-10-03 11:43:42 -04:00
65e585e122
Pause only if we're running low on queues to persist.
2020-10-03 11:43:34 -04:00
2ba36ee04c
Add a gauge for worker thread liveness.
2020-10-03 11:43:34 -04:00
fc05529574
Let MessagePersister manage its own worker thread.
2020-10-03 11:43:34 -04:00
07d24f487a
Don't re-register metrics for shared circuit breakers.
2020-10-02 15:05:00 -04:00
811acdb7f5
Use separate namespaces for Redis breaker/retry metrics.
2020-10-02 10:57:05 -04:00
a7266364d1
Refactor peer pruning to be more retry-friendly.
2020-10-01 17:17:07 -04:00
e83b41dc01
Reduce default Redis cluster command timeout to 3 seconds.
2020-10-01 17:17:07 -04:00
76665dd56e
Retry Redis commands that time out.
2020-10-01 17:17:07 -04:00
2d42b478ba
Consolidate cluster and pub/sub circuit breakers.
2020-10-01 17:17:07 -04:00
885fa6beae
Add tests for Device#isEnabled.
2020-10-01 12:54:35 -04:00
65cdd5fcbe
Drop the 365-day check when deciding if an account is enabled.
2020-10-01 12:54:35 -04:00
4302e19aba
Register a UUID argument factory for the messages database.
2020-10-01 11:06:43 -04:00
0c6f05f34a
Add a (failing!) test for sending a sealed-sender message after a non-sealed-sender message.
2020-10-01 11:06:43 -04:00
8040c285cd
Include stack traces when reporting persistence issues.
2020-09-30 11:47:16 -04:00
ada454f56f
Add a meter for persisting individual messages.
2020-09-30 10:39:56 -04:00
57d2ef8740
Return queues to the "to persist" list if something goes wrong during persistence.
2020-09-30 10:39:56 -04:00
a97e0982e3
Add an integration test for message persistence.
2020-09-30 10:39:56 -04:00
eaa2060d84
Fix an incorrect locking key and some previously-suppressed lock contention issues.
2020-09-30 10:39:56 -04:00
3e02c574e7
Log exceptions when persisting messages.
2020-09-30 10:39:56 -04:00
c7230ccbb0
Remove messages from the cache in bulk.
2020-09-29 10:58:02 -04:00
fc71ced660
Persist messages in batches.
2020-09-29 10:58:02 -04:00
6041a9d094
Make exit conditions slightly more conservative.
2020-09-29 10:58:02 -04:00
599cd766e1
Let Dropwizard manage persister thread lifecycles.
2020-09-29 10:58:02 -04:00
e64c8007c0
Detect GV2 capability in non-gcm Android devices
2020-09-28 15:54:10 -04:00
9339823e84
Add temporary metrics to monitor the ratio of enabled/disabled accounts.
2020-09-28 15:33:52 -04:00
e6d4620af1
Only allow linking desktop clients if they support the third-generation GV2 capability.
2020-09-25 17:08:32 -04:00
656e6db846
Only consider desktop devices GV2-capable if they send the third-gen GV2 capability.
2020-09-25 17:08:32 -04:00
30474e3a2b
Add a test for message ordering.
2020-09-25 11:41:58 -04:00
460bd98f1b
Add metrics for messages missing GUIDs.
2020-09-25 11:41:22 -04:00
a553eba574
Add an API endpoint for deleting accounts.
2020-09-25 11:39:17 -04:00
61f515670c
Add plumbing for deleting accounts and all associated data.
2020-09-25 11:39:17 -04:00
789af0f8a6
Add support for deleting keys associated with an account.
2020-09-25 11:39:17 -04:00
86fae58c96
Add support for deleting account entities from the database.
2020-09-25 11:39:17 -04:00
c54d3abe47
Check for the second-gen GV2 capability when linking devices.
2020-09-24 19:04:02 -04:00
6fe511eb50
Fix a bad size check when loading stored messages.
2020-09-23 18:02:33 -04:00
17d18b22c7
Drop pub/sub sending logic from WebsocketSender.
2020-09-23 14:51:02 -04:00
66a04ed730
Don't explicitly notify clients when messages get persisted.
2020-09-23 14:51:02 -04:00
7e14a0bc30
Drop pub/sub operations from WebsocketConnection.
2020-09-23 14:51:02 -04:00
77de0f86dc
Require desktop clients to send the new gv2-2 capability flag.
2020-09-23 12:05:58 -04:00
Jon Chambers