3cdc58200a
Copy headers from the initial websocket upgrade request into subsequent resource requests.
2021-03-10 14:54:11 -05:00
a1434524a4
Allow the storage service client to trust the Signal CA root.
2021-03-10 10:53:33 -05:00
07886a9722
Introduce a utility class for working with forwarding chains in HTTP headers.
2021-03-10 10:37:10 -05:00
fde1b49729
Introduce a set of gauges for various network statistics as reported by `nstat`.
2021-03-09 11:45:53 -05:00
af2a8548c3
Use Durations everywhere, drop unused constructors, and add tests.
2021-03-05 12:47:36 -05:00
f57a4171ba
Gather IP-based metrics for international, unsealed-sender messages.
2021-03-05 11:54:01 -05:00
eb6fe11da1
Add tools to decline messages from senders meeting specific conditions
2021-02-24 18:09:26 -05:00
5354104128
Only apply unsealed sender rate limits to targeted country codes.
2021-02-23 14:17:16 -05:00
b5ade5dc12
Only return payment address from latest profile ( #408 )
...
* Only return payment address from latest profile
* Rename `currentVersionedProfile` to `currentProfileVersion`
* Change return type to Optional
* Update service/src/main/java/org/whispersystems/textsecuregcm/controllers/ProfileController.java
Co-authored-by: Jon Chambers <63609320+jon-signal@users.noreply.github.com>
* Fix broken test
Co-authored-by: Jon Chambers <63609320+jon-signal@users.noreply.github.com>
2021-02-23 12:29:47 -06:00
06ca5f14fc
Record the age of accounts that send unsealed-sender messages.
2021-02-23 12:16:45 -05:00
f85c6bf828
Demonstrate JUnit 5 works by migrating an existing test
2021-02-19 13:30:44 -06:00
93f845610d
Remove payments list from Account
2021-02-19 13:18:30 -06:00
aa8525385a
Temporarily support `IncomingMessage.online` ( #404 )
...
iOS versions prior to 5.5.0.7 send `online` on `IncomingMessage`, rather
than on the top-level entity. This adds a temporary server-side adaptation,
to prevent client-side issues, like persistent typing indicators.
2021-02-18 12:21:16 -06:00
142376f360
Add payment address to the versioned profile
2021-02-16 16:12:50 -06:00
2dbab70c8c
Create utility endpoint for currency conversion
2021-02-16 11:57:34 -08:00
5f49772ca6
Control enforcement of unsealed sender rate limits via dynamic configuration.
2021-02-12 15:43:05 -05:00
ff448950ed
Collapse the feature flag system into the dynamic config system.
2021-02-12 15:23:34 -05:00
ad32555cc9
Drop the old push scheduler Redis singleton.
2021-02-11 14:01:11 -05:00
be8a1acca9
Remove message database from the codebase ( #395 )
...
* Remove message database from the codebase
* Remove unused ExperimentEnrollmentManager in test
* Be more stylish
2021-02-11 10:50:03 -06:00
e0ed8fa0b8
Introduce a hyper-log-log-based cardinality rate limiter
2021-02-11 10:36:26 -05:00
dcbf285fae
Un-ignore message controller tests.
2021-02-11 10:14:48 -05:00
71510a8199
Decommission the old directory cache.
2021-02-10 11:02:11 -05:00
2f105ed0a4
Add support for remote client deprecation
2021-02-09 12:24:44 -05:00
6c0de89de8
Remove scary message-throttling feature flags.
2021-02-08 11:55:01 -05:00
aa99e202b4
Clarify behavioral contract of the pre-key store
2021-02-08 11:45:57 -05:00
04728ea4bc
Drop the old Postgres-based pre-key store.
2021-02-08 11:40:43 -05:00
d09b36b1d5
Fix deprecated imports for Java6Assertions and Matchers
2021-02-08 09:45:33 -06:00
a5dd4f5fac
Remove signaling key from Device and AccountAttributes
2021-02-08 09:45:33 -06:00
3298db8683
Split traffic between the Postgres and Dynamo pre-key stores.
2021-02-04 15:18:53 -05:00
d4d9403829
Add a Dynamo-backed key store.
2021-02-04 15:18:53 -05:00
5e1a572bd8
Track impact of unsealed sender rate limits ( #374 )
2021-02-03 08:58:47 -08:00
0dcb4b645c
Build Dynamo DB backed Message Store ( #358 )
...
* Work in progress...
* Finish first pass draft of MessagesDynamoDb
* Use begins_with everywhere for destination device id
* Remove now unused methods
* First basic test built
* Add another test case
* Remove comment
* Verify more of the message contents
* Ensure all methods are tested
* Integrate MessagesDynamoDb into the MessagesManager
This change plugs the MessagesDynamoDb class into the live serving
flow in MessagesManager.
Tests are not yet as comprehensive for this big a change as they
should be, but they now compile and pass so checkpointing here with a
commit.
* Put DynamoDB before RDBS when deleting specific messages
* Extract method
* Make aws sdk version into a property
* Rename clientBuilder
* Discard messages with no GUID
* Unify batching logic into one function
* Comment on the source of the value in this constant
* Inline method
* Variable name swizzle
* Add timers to all public methods
* Add missing return statements
* Reject messages that are too large with response code 413
* Add configuration to control dynamo DB timeouts
* Set server timestamp from the ReceiptSender
* Change to shorter key names to optimize IOPS
* Fix tests broken by changing column names
* Fix broken copyright template output
* Remove copyright template error text
* Add experiments to control use of dynamo and rds in message storage
* Specify instance profile credentials for the dynamic configuration manager
* Use property for aws sdk version
* Switch dynamo to instance profile credentials
* Add metrics to the batch write loop
* Use placeholders in logging
2021-02-03 10:03:19 -06:00
1a27c7eabc
Add a (failing) test for new Ivory Coast phone numbers.
2021-02-01 21:25:14 -05:00
408b959441
Require a push challenge when registering (or else require a captcha).
2021-02-01 20:44:21 -05:00
35fc98a188
Add an experiment enrollment manager.
2021-02-01 11:08:16 -05:00
92f6a79e1f
Add a dynamic configuration manager
2021-02-01 11:01:58 -05:00
8f94ed68a3
Ignore expired devices when checking for GV1->GV2 migration capability.
2021-01-30 16:55:05 -05:00
92a0deffcf
Add more robust tests for about/emoji fields.
2021-01-27 10:34:13 -05:00
24ea6a9f1d
Revert "Temporarily disable registration abuse system"
...
This reverts commit 22ef058cb6
2021-01-25 14:58:27 -05:00
f10be893ce
Drop the old feature flag controller.
2021-01-25 14:55:57 -05:00
c606c1664f
Add admin tasks for listing, setting, and deleting feature flags.
2021-01-25 14:55:57 -05:00
225932b4c9
Add emojis/"about" text to profiles
2021-01-20 15:42:47 -05:00
6b850b9894
Allow (versioned) profile names up to 380 base64 characters long.
2021-01-20 11:08:10 -05:00
943a5d1036
Shard push scheduling cache
2021-01-19 15:50:12 -05:00
0a23ce870a
Allow message persisters to be disabled by a feature flag.
2021-01-17 11:13:12 -05:00
a57ce1dd17
Add machinery to allow a percentage of message sends to succeed.
2021-01-15 17:05:16 -05:00
81c1ba6eef
Respond to all "message send" attempts with HTTP/503.
2021-01-15 15:34:14 -05:00
9c53d818f4
Use separate clusters for message cache read/write operations.
2021-01-15 13:51:39 -05:00
e5a2c1ab10
Always return an empty list of prekeys.
2021-01-15 12:27:10 -05:00
67ed035b36
Retry serializable key transactions.
2021-01-13 17:38:29 -05:00
2e01da5ec1
Add a task to enable/disable accelerated crawling.
2021-01-11 19:29:18 -05:00
9412a7424c
Return HTTP/429 whenever somebody tries to get contacts from the old directory system.
2021-01-11 19:29:10 -05:00
f8cbb4f386
Temporarily suspend client version metrics to reduce load on our metric aggregator.
2021-01-11 14:04:44 -05:00
86ccaa52a5
Allow configuration of multiple directory account crawler listeners ( #325 )
...
* Allow configuration of multiple directory account crawler listeners
Only one should update the local redis directory. This one is marked
with replicationPrimary true. The others in the list only serve to
issue replication requests over to CDS replication load balancers.
* Update one more metric name
2021-01-10 17:11:02 -06:00
1b53f10091
Reload scripts across the whole cluster if one shard is missing the script.
2021-01-10 15:00:12 -05:00
bac268a21c
Don't send a reply to clients until messages are safely in a non-volatile store.
2021-01-10 13:03:40 -05:00
22ef058cb6
Temporarily disable registration abuse system
2021-01-09 15:57:55 -05:00
9ee6419bc0
Publish directory updates to multiple SQS queues.
2021-01-08 18:07:18 -05:00
3bf0188e7f
Turn off alphanumeric sender ID for all countries.
2021-01-08 06:18:53 -05:00
91fc0fd623
Revert "Delete data in the storage service when deleting accounts."
...
This reverts commit ff1a721d5b
2021-01-08 06:18:39 -05:00
a4d0c17efd
Record OS versions for iOS requests.
2020-12-23 11:36:31 -05:00
ff1a721d5b
Delete data in the storage service when deleting accounts.
2020-12-23 11:35:38 -05:00
c870a1bbd5
Introduce a storage service client.
2020-12-23 11:35:38 -05:00
e551fd2c1b
Revert "Pause checks for GV1 migration when checking for capability downgrades."
...
This reverts commit e7745db36e
2020-12-10 17:02:41 -05:00
34a11c2338
Record OS versions for desktop and SDK versions for Android.
2020-12-10 17:02:05 -05:00
e524ff965d
Add a utility method for getting client platform tags from UA strings for metrics.
2020-12-10 17:01:46 -05:00
92fde83b3a
Discard oversized messages bound for desktop clients via websockets.
2020-12-07 15:03:35 -05:00
299b680013
Always include UUID in UD certificate ( #300 )
2020-12-01 08:56:55 -06:00
00a3e562dc
Force use of UCS-2 instead of GSM-7 for SMS to China ( #297 )
2020-11-20 14:41:48 -06:00
e7745db36e
Pause checks for GV1 migration when checking for capability downgrades.
2020-11-17 09:25:12 -05:00
474b879b16
Only notify CDS if an account attribute change actually changes an account's discoverability.
2020-11-16 10:54:12 -05:00
325e65db7f
Expand UA parsing tests to cover OS details in desktop strings.
2020-11-11 13:05:18 -05:00
604287244f
Update copyright statement on all source files
...
IntelliJ Copyright Profile used to automate this.
2020-11-04 11:55:35 -05:00
5587b7d469
Expose gv1-migration on profile endpoint
2020-10-28 13:00:57 -04:00
26870d134f
Set source UUID when delivering envelopes from message cache/db on websocket
2020-10-28 12:38:32 -04:00
c2db2d3cbd
Add GV1 Migration capability
2020-10-27 16:17:21 -04:00
05d9ec673e
Send push notifications if websockets close before all messages are delivered
2020-10-27 16:02:55 -04:00
96d3a69479
Use container-managed executors for APN/GCM senders.
2020-10-23 11:30:03 -04:00
23ca011ac1
Record account deletion reasons.
2020-10-20 19:00:34 -04:00
99f488d48f
Drop websocket connection names (unused for a while now).
2020-10-19 11:24:35 -04:00
05929871c9
Rename PushSender to MessageSender and add docs.
2020-10-19 11:24:35 -04:00
74b3daa70a
Collapse WebsocketSender into PushSender.
2020-10-19 11:24:35 -04:00
3a84775912
Log cluster topology change events, too.
2020-10-13 16:07:08 -04:00
a9d0aa136d
Add OS-reported metrics for cached/buffered memory.
2020-10-11 13:43:15 -04:00
39c09733d3
Add /v1/payments/auth endpoint
2020-10-08 10:51:01 -04:00
e1c397993d
Require Android clients to support the gv2-3 capability
2020-10-06 16:49:49 -04:00
2b6811cb1b
Really delete old accounts instead of just removing their push channels.
2020-10-06 11:04:47 -04:00
c82496b972
Remove the "repair queue metadata" script.
2020-10-05 16:57:16 -04:00
bb087caddc
Don't panic if a queue exists, but is empty when repairing metadata.
2020-10-04 16:09:56 -04:00
5e3f8b9c2e
Disallow insertion of duplicate messages.
2020-10-04 15:34:14 -04:00
1ccfe928f7
Add a test to make sure that we don't double-insert messages with the same GUID.
2020-10-04 15:34:14 -04:00
952cfae4e6
Repair queue metadata before persisting queues.
2020-10-04 15:25:06 -04:00
02a2c3224f
Discard unused feature flag constants/mocking.
2020-10-04 10:48:42 -04:00
fc05529574
Let MessagePersister manage its own worker thread.
2020-10-03 11:43:34 -04:00
76665dd56e
Retry Redis commands that time out.
2020-10-01 17:17:07 -04:00
885fa6beae
Add tests for Device#isEnabled.
2020-10-01 12:54:35 -04:00
65cdd5fcbe
Drop the 365-day check when deciding if an account is enabled.
2020-10-01 12:54:35 -04:00
0c6f05f34a
Add a (failing!) test for sending a sealed-sender message after a non-sealed-sender message.
2020-10-01 11:06:43 -04:00
57d2ef8740
Return queues to the "to persist" list if something goes wrong during persistence.
2020-09-30 10:39:56 -04:00
a97e0982e3
Add an integration test for message persistence.
2020-09-30 10:39:56 -04:00
eaa2060d84
Fix an incorrect locking key and some previously-suppressed lock contention issues.
2020-09-30 10:39:56 -04:00
c7230ccbb0
Remove messages from the cache in bulk.
2020-09-29 10:58:02 -04:00
fc71ced660
Persist messages in batches.
2020-09-29 10:58:02 -04:00
599cd766e1
Let Dropwizard manage persister thread lifecycles.
2020-09-29 10:58:02 -04:00
e64c8007c0
Detect GV2 capability in non-gcm Android devices
2020-09-28 15:54:10 -04:00
e6d4620af1
Only allow linking desktop clients if they support the third-generation GV2 capability.
2020-09-25 17:08:32 -04:00
656e6db846
Only consider desktop devices GV2-capable if they send the third-gen GV2 capability.
2020-09-25 17:08:32 -04:00
30474e3a2b
Add a test for message ordering.
2020-09-25 11:41:58 -04:00
a553eba574
Add an API endpoint for deleting accounts.
2020-09-25 11:39:17 -04:00
61f515670c
Add plumbing for deleting accounts and all associated data.
2020-09-25 11:39:17 -04:00
789af0f8a6
Add support for deleting keys associated with an account.
2020-09-25 11:39:17 -04:00
86fae58c96
Add support for deleting account entities from the database.
2020-09-25 11:39:17 -04:00
66a04ed730
Don't explicitly notify clients when messages get persisted.
2020-09-23 14:51:02 -04:00
7e14a0bc30
Drop pub/sub operations from WebsocketConnection.
2020-09-23 14:51:02 -04:00
77de0f86dc
Require desktop clients to send the new gv2-2 capability flag.
2020-09-23 12:05:58 -04:00
e9e18afb4a
Add a (failing) integration test demonstrating an infinite loop.
2020-09-22 10:21:33 -04:00
62c31eb202
Revert "Revert keyspace delivery for all messages"
...
This reverts commit 4dc49604b6
2020-09-22 10:21:33 -04:00
5986145282
Add a second-generation GV2 capability and ignore the old capability for iOS devices.
2020-09-21 18:57:53 -04:00
83f9eacac4
Refactor UserAgentTagUtil to parse UA strings with UserAgentUtil.
2020-09-21 12:24:08 -04:00
baab6b951b
Add a general utility class for parsing user-agent strings.
2020-09-21 12:24:08 -04:00
ebc3a251b7
Drop the UUID addressing capability flag entirely.
2020-09-14 15:36:29 -04:00
4dc49604b6
Revert keyspace delivery for all messages
...
* Revert "Send all messages via keyspace notifications when a feature flag is enabled."
This reverts commit fadcf62166c02b25576689788fa665a052e2ee8f158e5004b76f9ff3be37f766c577438f53152c3e7bbc88d71668256d2343
2020-09-14 15:35:10 -04:00
fadcf62166
Send all messages via keyspace notifications when a feature flag is enabled.
2020-09-11 13:12:17 -04:00
a052e2ee8f
Refactor: collapse state into semaphores/atomic booleans.
2020-09-11 13:12:17 -04:00
6f9ff3be37
Avoid querying the database if we think all new messages are in the cache.
2020-09-11 13:12:17 -04:00
f766c57743
Query for more stored messages if an update happens while we're already processing a batch.
2020-09-11 13:12:17 -04:00
8f53152c3e
Only send the "queue cleared" message once per websocket session.
2020-09-11 13:12:17 -04:00
7bbc88d716
Let processStoredMessages handle requery logic.
2020-09-11 13:12:17 -04:00
68256d2343
Only allow one thread to process stored messages at a time.
2020-09-11 13:12:17 -04:00
f88c440c48
Automatically retry when Twilio returns unreachable ( #190 )
...
* Parse and log the Twilio error code
* Automatically retry without sender ID when Twilio returns unreachable
* Remove attempt count and pass around whether or not sender id was used
2020-09-10 13:58:39 -05:00
2c6b646d87
Enforce no capability downgrade on device verification
2020-09-09 16:05:00 -04:00
e7572094b5
Require all enabled devices to support GV2.
2020-09-09 16:05:00 -04:00
fdef21a871
Record and listen for ephemeral messages in a separate queue.
2020-09-09 14:42:09 -04:00
d40cff8a99
Revert "Add a system for storing, retrieving, and notifying listeners about ephemeral (online) messages."
...
This reverts commit 06754d6158
2020-09-08 15:55:09 -04:00
06754d6158
Add a system for storing, retrieving, and notifying listeners about ephemeral (online) messages.
2020-09-08 11:14:42 -04:00
1d5087374e
Jettison UUID-or-E164 plumbing in favor of UUID-only.
2020-09-08 09:30:47 -04:00
8356264fe0
Rename RedisClusterMessagesCache and related classes to just MessagesCache.
2020-09-08 09:30:47 -04:00
18ecd748dd
Entirely discard the old message cache machinery.
2020-09-08 09:30:47 -04:00
5c04f2634a
Use a dedicated executor service for dispatching keyspace notifications.
2020-09-03 11:04:48 -04:00
81e8143a43
Rely solely on the clustered message cache.
2020-09-02 11:57:33 -04:00
8409986ef5
Mirror persistence operations from the new persister to the old persister.
2020-09-02 11:02:40 -04:00
2b50367d7f
Put message persisters behind feature flags.
2020-09-02 11:02:40 -04:00
5221828705
Increase maximum sticker size to 300 kibibytes
...
In preparation for animated stickers, allow stickers to be up to 300
kibibytes.
2020-09-01 10:55:26 -04:00
ce026e7ad0
Don't send contacts to CDS if they've opted out of discoverability. (SERVER-130)
2020-08-27 15:58:02 -04:00
58e3122dab
Add a discoverableByPhoneNumber account attribute. (SERVER-129)
2020-08-27 15:58:02 -04:00
32b18c9509
Add an endpoint for getting the current state of feature flags.
2020-08-27 13:18:12 -04:00
acf52ad8a3
Make feature flag manager tests use a real database to avoid over-mocking.
2020-08-27 13:18:12 -04:00
08dd493f98
Don't report exceptions as part of traffic metrics.
2020-08-27 13:17:57 -04:00
07bbe7dfb2
Return to an async model for push notification latency.
2020-08-27 10:51:44 -04:00
0aa1b80e3e
Add a command for persisting any detached messages in the old message cache.
2020-08-27 10:51:12 -04:00
Jon Chambers