635f669a32
Count slow queue drain events by platform.
2021-02-12 15:43:50 -05:00
5f49772ca6
Control enforcement of unsealed sender rate limits via dynamic configuration.
2021-02-12 15:43:05 -05:00
6332552346
Record the sender's country when we reject unsealed-sender messages.
2021-02-12 15:23:52 -05:00
4fb7afcf7b
Drop the feature flags table.
2021-02-12 15:23:34 -05:00
ff448950ed
Collapse the feature flag system into the dynamic config system.
2021-02-12 15:23:34 -05:00
d6319aeb92
just set OWA,OWP,OWI user agents upon token deletion
2021-02-12 15:22:50 -05:00
8fc6f9c442
when a user unregisters, record his user agent for posterity
2021-02-12 15:22:50 -05:00
e9ea79cc8e
Shorten eviction time to 7 days (to match message retention time).
2021-02-11 14:01:11 -05:00
ad32555cc9
Drop the old push scheduler Redis singleton.
2021-02-11 14:01:11 -05:00
be8a1acca9
Remove message database from the codebase ( #395 )
...
* Remove message database from the codebase
* Remove unused ExperimentEnrollmentManager in test
* Be more stylish
2021-02-11 10:50:03 -06:00
e0ed8fa0b8
Introduce a hyper-log-log-based cardinality rate limiter
2021-02-11 10:36:26 -05:00
dcbf285fae
Un-ignore message controller tests.
2021-02-11 10:14:48 -05:00
ceda459942
Substitute some !Optional.isPresent with Optional.isEmpty
2021-02-10 13:21:44 -06:00
28fe44aea4
Remove messages dynamo db experiment
2021-02-10 10:21:49 -06:00
71510a8199
Decommission the old directory cache.
2021-02-10 11:02:11 -05:00
9cd121c8f6
Record initial queue drain times faceted by client platform.
2021-02-09 19:21:46 -05:00
2f105ed0a4
Add support for remote client deprecation
2021-02-09 12:24:44 -05:00
704d54dd01
Record a distribution of the number of keys available when checked.
2021-02-08 16:50:27 -05:00
bee9b61831
Record a distribution of initial queue depths broken down by platform.
2021-02-08 16:06:21 -05:00
9c6ce08db0
Break down message send attempts by client platform, message ephemerality, and sealed sender status.
2021-02-08 16:06:04 -05:00
6c0de89de8
Remove scary message-throttling feature flags.
2021-02-08 11:55:01 -05:00
aa99e202b4
Clarify behavioral contract of the pre-key store
2021-02-08 11:45:57 -05:00
04728ea4bc
Drop the old Postgres-based pre-key store.
2021-02-08 11:40:43 -05:00
6865cdfce3
Replace all org.hibernate validators with modern javax.validation
2021-02-08 09:45:33 -06:00
d09b36b1d5
Fix deprecated imports for Java6Assertions and Matchers
2021-02-08 09:45:33 -06:00
a5dd4f5fac
Remove signaling key from Device and AccountAttributes
2021-02-08 09:45:33 -06:00
9936b2967e
Don't use signaling key anymore
...
Signaling key was deprecated over 2 years ago. It's time for it to go.
2021-02-08 09:45:33 -06:00
0971613ac0
Remove messages table from account DB
...
This is quite out of date. Messages has been a separate database for a
long long time.
2021-02-08 09:45:33 -06:00
f7d855c59e
Drop unsealed sender rate limit logging from `info` to `debug`.
2021-02-05 11:43:37 -05:00
2fe743649d
Extract common UUID operations into a utility class.
2021-02-05 10:40:27 -05:00
a015237fd2
Don't request data from DynamoDB if we already have it locally.
2021-02-05 10:40:27 -05:00
e1f4deaacc
Add a generalized countItemsMatchingQuery method for DynamoDB stores.
2021-02-05 10:40:27 -05:00
1dceee3fa0
Make DynamoDBConfiguration more consistent.
2021-02-05 10:40:27 -05:00
3a17a7c98f
Explicitly make the dynamic config worker a daemon thread.
2021-02-04 19:05:51 -05:00
3298db8683
Split traffic between the Postgres and Dynamo pre-key stores.
2021-02-04 15:18:53 -05:00
d4d9403829
Add a Dynamo-backed key store.
2021-02-04 15:18:53 -05:00
426e6923ac
Continue to poll for config changes after startup.
2021-02-04 13:46:37 -05:00
b413f665d8
Make DynamicConfigurationManager not-really-Managed.
2021-02-04 11:33:33 -05:00
5e1a572bd8
Track impact of unsealed sender rate limits ( #374 )
2021-02-03 08:58:47 -08:00
0dcb4b645c
Build Dynamo DB backed Message Store ( #358 )
...
* Work in progress...
* Finish first pass draft of MessagesDynamoDb
* Use begins_with everywhere for destination device id
* Remove now unused methods
* First basic test built
* Add another test case
* Remove comment
* Verify more of the message contents
* Ensure all methods are tested
* Integrate MessagesDynamoDb into the MessagesManager
This change plugs the MessagesDynamoDb class into the live serving
flow in MessagesManager.
Tests are not yet as comprehensive for this big a change as they
should be, but they now compile and pass so checkpointing here with a
commit.
* Put DynamoDB before RDBS when deleting specific messages
* Extract method
* Make aws sdk version into a property
* Rename clientBuilder
* Discard messages with no GUID
* Unify batching logic into one function
* Comment on the source of the value in this constant
* Inline method
* Variable name swizzle
* Add timers to all public methods
* Add missing return statements
* Reject messages that are too large with response code 413
* Add configuration to control dynamo DB timeouts
* Set server timestamp from the ReceiptSender
* Change to shorter key names to optimize IOPS
* Fix tests broken by changing column names
* Fix broken copyright template output
* Remove copyright template error text
* Add experiments to control use of dynamo and rds in message storage
* Specify instance profile credentials for the dynamic configuration manager
* Use property for aws sdk version
* Switch dynamo to instance profile credentials
* Add metrics to the batch write loop
* Use placeholders in logging
2021-02-03 10:03:19 -06:00
fc4c8d6054
Update to the latest version of libphonenumber.
2021-02-01 21:25:14 -05:00
1a27c7eabc
Add a (failing) test for new Ivory Coast phone numbers.
2021-02-01 21:25:14 -05:00
408b959441
Require a push challenge when registering (or else require a captcha).
2021-02-01 20:44:21 -05:00
35fc98a188
Add an experiment enrollment manager.
2021-02-01 11:08:16 -05:00
92f6a79e1f
Add a dynamic configuration manager
2021-02-01 11:01:58 -05:00
8f94ed68a3
Ignore expired devices when checking for GV1->GV2 migration capability.
2021-01-30 16:55:05 -05:00
ce1a4b94cb
Actually store emoji/about text in the database.
2021-01-27 10:34:13 -05:00
92a0deffcf
Add more robust tests for about/emoji fields.
2021-01-27 10:34:13 -05:00
97b6f6028b
Fix a minor typo in the help text for a feature flag task.
2021-01-25 18:03:38 -05:00
611e8c39ee
Actually drop feature flag config.
2021-01-25 15:20:06 -05:00
01f1c263a6
Add a meter for captcha requests.
2021-01-25 14:58:27 -05:00
24ea6a9f1d
Revert "Temporarily disable registration abuse system"
...
This reverts commit 22ef058cb6
2021-01-25 14:58:27 -05:00
46c800b8b7
Smoosh request logging tasks together rather than having one task for each direction.
2021-01-25 14:58:15 -05:00
f10be893ce
Drop the old feature flag controller.
2021-01-25 14:55:57 -05:00
c606c1664f
Add admin tasks for listing, setting, and deleting feature flags.
2021-01-25 14:55:57 -05:00
225932b4c9
Add emojis/"about" text to profiles
2021-01-20 15:42:47 -05:00
6b850b9894
Allow (versioned) profile names up to 380 base64 characters long.
2021-01-20 11:08:10 -05:00
943a5d1036
Shard push scheduling cache
2021-01-19 15:50:12 -05:00
b25da8ceaa
Don't attempt SMS to iran ( #355 )
2021-01-19 09:13:37 -08:00
10cdb7387d
Be consistent with use of DataSize class
2021-01-18 17:01:43 -06:00
dd436dd1dd
Create a Meter for tracking messages larger than 256kib
2021-01-18 17:01:43 -06:00
13b84635b5
Drop an unused message database index. ( #352 )
2021-01-18 10:26:03 -06:00
27534d408f
Log when messages cache detects topology change ( #354 )
...
Co-authored-by: Moxie Marlinspike <moxie+github@signal.org>
2021-01-17 17:13:23 -08:00
0a23ce870a
Allow message persisters to be disabled by a feature flag.
2021-01-17 11:13:12 -05:00
c355ef8d53
Reduce the message cache thread pool size.
2021-01-16 11:15:25 -05:00
1feb23ba99
Stop periodic topology refreshes.
2021-01-16 03:35:36 -05:00
59a0fd0799
Embiggen message cluster thread pool.
2021-01-16 02:57:04 -05:00
00b5cfcf17
Allow the client presence manager to use an entirely separate cluster.
2021-01-16 02:57:04 -05:00
9e342f253d
Use the same client for inserts and reads in the message cache cluster.
2021-01-16 01:50:40 -05:00
20c48b6bb2
Expand message-related thread pools to 1 thread per shard.
2021-01-16 01:50:40 -05:00
4f9e7bb572
Separate Lettuce thread pools.
2021-01-16 01:18:05 -05:00
0a322d5a9f
Add a "doomsday switch."
2021-01-15 18:05:18 -05:00
59eb6d10c1
Gate based on destination rather than random.
2021-01-15 18:05:18 -05:00
a57ce1dd17
Add machinery to allow a percentage of message sends to succeed.
2021-01-15 17:05:16 -05:00
b100b3c36b
Reject traffic without logging exceptions
2021-01-15 16:23:53 -05:00
81c1ba6eef
Respond to all "message send" attempts with HTTP/503.
2021-01-15 15:34:14 -05:00
93ae4d1ee6
Move the client presence manager to its own breaker.
2021-01-15 13:51:39 -05:00
9c53d818f4
Use separate clusters for message cache read/write operations.
2021-01-15 13:51:39 -05:00
e5a2c1ab10
Always return an empty list of prekeys.
2021-01-15 12:27:10 -05:00
67ed035b36
Retry serializable key transactions.
2021-01-13 17:38:29 -05:00
ad30786f4a
Parallelize message persisters.
2021-01-12 18:50:14 -05:00
2e01da5ec1
Add a task to enable/disable accelerated crawling.
2021-01-11 19:29:18 -05:00
8fb37a0024
Log when a crawling cycle has wrapped up.
2021-01-11 19:29:18 -05:00
9412a7424c
Return HTTP/429 whenever somebody tries to get contacts from the old directory system.
2021-01-11 19:29:10 -05:00
f8cbb4f386
Temporarily suspend client version metrics to reduce load on our metric aggregator.
2021-01-11 14:04:44 -05:00
86ccaa52a5
Allow configuration of multiple directory account crawler listeners ( #325 )
...
* Allow configuration of multiple directory account crawler listeners
Only one should update the local redis directory. This one is marked
with replicationPrimary true. The others in the list only serve to
issue replication requests over to CDS replication load balancers.
* Update one more metric name
2021-01-10 17:11:02 -06:00
cc3e5d23e4
Enable Lettuce adaptive topology refreshes.
2021-01-10 16:20:35 -05:00
cac86d1f77
Standardize toplogy event handling strategy.
2021-01-10 15:14:12 -05:00
22f7bb822f
Raise log level of toplogy changes.
2021-01-10 15:14:12 -05:00
1b53f10091
Reload scripts across the whole cluster if one shard is missing the script.
2021-01-10 15:00:12 -05:00
bac268a21c
Don't send a reply to clients until messages are safely in a non-volatile store.
2021-01-10 13:03:40 -05:00
321e6e6679
Don't validate cluster membership (allow new shards to join dynamically).
2021-01-10 12:58:35 -05:00
22ef058cb6
Temporarily disable registration abuse system
2021-01-09 15:57:55 -05:00
9ee6419bc0
Publish directory updates to multiple SQS queues.
2021-01-08 18:07:18 -05:00
3bf0188e7f
Turn off alphanumeric sender ID for all countries.
2021-01-08 06:18:53 -05:00
91fc0fd623
Revert "Delete data in the storage service when deleting accounts."
...
This reverts commit ff1a721d5b
2021-01-08 06:18:39 -05:00
d2fcf68381
Record the status message when clients reject websocket messages.
2020-12-23 12:29:15 -05:00
a4d0c17efd
Record OS versions for iOS requests.
2020-12-23 11:36:31 -05:00
ff1a721d5b
Delete data in the storage service when deleting accounts.
2020-12-23 11:35:38 -05:00
c870a1bbd5
Introduce a storage service client.
2020-12-23 11:35:38 -05:00
Jon Chambers