df01be2dca
Don’t throw exceptions from ReportMessageManager#store()
2021-05-13 18:33:02 -05:00
e320626c6e
Add report message API
2021-05-13 17:19:34 -05:00
bfd2c32d4e
Add sender key capability
2021-05-12 18:15:25 -05:00
e9a3d52d7f
Add an optional description to the payment intent request
2021-05-12 18:14:33 -05:00
d45154f2aa
Measure captcha challenge success rates.
2021-05-12 12:20:53 -05:00
760462f8fb
Add configuration for regional SMS verification text
2021-05-12 12:20:46 -05:00
46110d4d65
Add client challenges for prekey and message rate limiters
2021-05-11 16:21:32 -05:00
17047513c3
Create stripe api endpoint for apple pay donations
2021-05-11 10:03:04 -05:00
350de1c759
Add a simple utility class for testing if an IP belongs to a Tor exit node.
2021-05-06 11:57:18 -04:00
8fdbcbef44
Send directory updates in batches.
2021-05-04 15:18:43 -04:00
fac4538f6f
Migrate rate limiters to rate limiter cluster
2021-05-03 10:57:34 -05:00
4f6b132449
Add secondaryCacheCluster to RateLimiter
2021-04-30 15:26:17 -05:00
e4dbb8efe7
Fix deserialization for Duration in dynamic config
2021-04-28 15:57:23 -05:00
59e401f41e
Use a MIME Base64 decoder for attachment signing keys.
2021-04-26 18:53:15 -04:00
6196856a7c
Use the JDK-provided Base64 encoder/decoder.
2021-04-26 18:17:03 -04:00
0e8d4f9a61
Drop Bouncy Castle as a dependency.
2021-04-26 17:58:19 -04:00
5ee56b022c
Add CustomFriendlyName parameter to Twilio Verify requests
2021-04-26 14:31:52 -05:00
6c37b658ac
Migrate VoiceVerificationControllerTest to JUnit 5
2021-04-26 14:31:52 -05:00
1f53900345
Migrate VoiceVerificationController to Util#findBestLocale
2021-04-26 14:31:52 -05:00
deece33a0d
Fix parameterized testWhoAmI
2021-04-26 14:31:52 -05:00
13053da97f
Add Twilio Verify experiment to AccountController
2021-04-26 14:31:52 -05:00
4c019aef15
Migrate PendingAccountsTest to JUnit 5
2021-04-26 14:31:52 -05:00
bab5e5769b
Add TwilioVerifyExperimentEnrollmentManager
2021-04-26 14:31:52 -05:00
76cbf734ad
Add TwilioVerifySender to TwilioSmsSender
2021-04-26 14:31:52 -05:00
17ba630014
Add TwilioVerifySender
2021-04-26 14:31:52 -05:00
f190462879
Fully implement unsealed sender cardinality rate limiter
2021-04-23 11:45:53 -05:00
6c6f073bc2
Pass re-migration account in test
2021-04-22 17:11:10 -05:00
abafa2ccac
Include ATTR_MIGRATION_VERSION in update()
2021-04-21 14:13:13 -05:00
5974328d9c
Ensure accounts are deleted after batch migration; store migration failures for later processing
2021-04-21 11:13:07 -05:00
a472774734
Add threadpoool to increase Accounts → Dynamo migration throughput
2021-04-21 11:13:07 -05:00
166d203e8e
Don’t PUT unmigrated accounts in update()
2021-04-21 11:13:07 -05:00
f2a1a65a45
Migrate MessageControllerTest to JUnit 5
2021-04-20 13:00:29 -05:00
52478e7de0
Test the account crawler against a real redis cluster.
2021-04-20 13:58:27 -04:00
59bbd0c43c
Add Accounts DynamoDB
...
* Add additional test cases to AccountsTest
* Migrate AccountsManagerTest to JUnit 5
* Add AccountsDynamoDbConfiguration
* Add Account.dynamoDbMigrationversion
* Add DynamicAccountsDynamoDbMigrationConfiguration
* Add AccountsDynamoDb to AccountsManager
* Add AccountsDynamoDbMigrator
2021-04-16 14:24:24 -05:00
1f24c913a6
add unit tests for SmsSender
2021-04-08 09:17:47 -04:00
2cb788ceb7
Delete secure backups when deleting accounts.
2021-04-01 11:30:48 -04:00
257fef9734
Add a secure backup service client.
2021-04-01 11:30:48 -04:00
ab4e94edab
Revert "Revert "Delete data in the storage service when deleting accounts.""
...
This reverts commit 91fc0fd623
2021-03-30 17:17:32 -04:00
ad6c271f9d
Add dynamic configuration for signup captcha by country code
2021-03-30 10:22:02 -05:00
91bd061110
Migrate deprecated Lettuce method and enum usages
2021-03-29 14:51:26 -05:00
3745a0b81d
Update from 684 to 776 for payment address length to account for signature
2021-03-27 00:45:47 -05:00
7e29ed1cc7
Block attempts to set wallet addresses from unsupported countries.
2021-03-26 12:21:11 -04:00
5965f0fd22
Add a dynamically-configured list of allowed country codes for payments.
2021-03-26 12:21:11 -04:00
a816aa0186
Revert "Add a storage client method for checking wheter a user has a stored manifest."
...
This reverts commit 8b6012f8a8
2021-03-22 10:32:38 -04:00
089b6b1644
Retry attempts to get messages after a delay; close connections after a finite number of retries.
2021-03-22 10:32:25 -04:00
e5ae0572c5
Add android-2021-03 verification format
2021-03-18 12:49:50 -05:00
63dac3bd9f
Migrate AccountControllerTest to JUnit 5
2021-03-18 12:45:20 -05:00
0bc1369e04
Work through the full list of supported locales when choosing a language for voice verification.
2021-03-18 13:29:09 -04:00
ca2f7d2eed
Parse locale strings when sending voice verification codes.
2021-03-18 13:29:09 -04:00
8b6012f8a8
Add a storage client method for checking wheter a user has a stored manifest.
2021-03-18 12:43:12 -04:00
ad838b4827
Add isEnrolled(e164, experiment) for pre-registration experiments
2021-03-16 13:09:10 -05:00
25f603efc9
Add DynamicPreRegistrationExperimentEnrollmentConfiguration
2021-03-16 13:09:10 -05:00
b5bd16c6a9
Migrate DynamicConfigurationTest to JUnit 5
2021-03-16 13:09:10 -05:00
14bfa83bb8
Migrate ExperimentEnrollmentManagerTest to JUnit 5
2021-03-16 13:09:10 -05:00
5dc8086968
Migrate Twilio numbers from static to dynamic configuration
2021-03-16 12:53:51 -05:00
efe7f2e4c1
Remove vestiges of per-country sender ID logic/configuration.
2021-03-10 14:58:46 -05:00
fb2fc2335a
Require messaging service IDs; remove fallback-to-random-number logic.
2021-03-10 14:58:46 -05:00
345e116699
Place our trust in a Twilio message service.
2021-03-10 14:58:46 -05:00
e50a1c0646
Revert "Turn off alphanumeric sender ID for all countries."
...
This reverts commit 3bf0188e7f
2021-03-10 14:58:46 -05:00
3cdc58200a
Copy headers from the initial websocket upgrade request into subsequent resource requests.
2021-03-10 14:54:11 -05:00
a1434524a4
Allow the storage service client to trust the Signal CA root.
2021-03-10 10:53:33 -05:00
07886a9722
Introduce a utility class for working with forwarding chains in HTTP headers.
2021-03-10 10:37:10 -05:00
fde1b49729
Introduce a set of gauges for various network statistics as reported by `nstat`.
2021-03-09 11:45:53 -05:00
af2a8548c3
Use Durations everywhere, drop unused constructors, and add tests.
2021-03-05 12:47:36 -05:00
f57a4171ba
Gather IP-based metrics for international, unsealed-sender messages.
2021-03-05 11:54:01 -05:00
eb6fe11da1
Add tools to decline messages from senders meeting specific conditions
2021-02-24 18:09:26 -05:00
5354104128
Only apply unsealed sender rate limits to targeted country codes.
2021-02-23 14:17:16 -05:00
b5ade5dc12
Only return payment address from latest profile ( #408 )
...
* Only return payment address from latest profile
* Rename `currentVersionedProfile` to `currentProfileVersion`
* Change return type to Optional
* Update service/src/main/java/org/whispersystems/textsecuregcm/controllers/ProfileController.java
Co-authored-by: Jon Chambers <63609320+jon-signal@users.noreply.github.com>
* Fix broken test
Co-authored-by: Jon Chambers <63609320+jon-signal@users.noreply.github.com>
2021-02-23 12:29:47 -06:00
06ca5f14fc
Record the age of accounts that send unsealed-sender messages.
2021-02-23 12:16:45 -05:00
f85c6bf828
Demonstrate JUnit 5 works by migrating an existing test
2021-02-19 13:30:44 -06:00
93f845610d
Remove payments list from Account
2021-02-19 13:18:30 -06:00
aa8525385a
Temporarily support `IncomingMessage.online` ( #404 )
...
iOS versions prior to 5.5.0.7 send `online` on `IncomingMessage`, rather
than on the top-level entity. This adds a temporary server-side adaptation,
to prevent client-side issues, like persistent typing indicators.
2021-02-18 12:21:16 -06:00
142376f360
Add payment address to the versioned profile
2021-02-16 16:12:50 -06:00
2dbab70c8c
Create utility endpoint for currency conversion
2021-02-16 11:57:34 -08:00
5f49772ca6
Control enforcement of unsealed sender rate limits via dynamic configuration.
2021-02-12 15:43:05 -05:00
ff448950ed
Collapse the feature flag system into the dynamic config system.
2021-02-12 15:23:34 -05:00
ad32555cc9
Drop the old push scheduler Redis singleton.
2021-02-11 14:01:11 -05:00
be8a1acca9
Remove message database from the codebase ( #395 )
...
* Remove message database from the codebase
* Remove unused ExperimentEnrollmentManager in test
* Be more stylish
2021-02-11 10:50:03 -06:00
e0ed8fa0b8
Introduce a hyper-log-log-based cardinality rate limiter
2021-02-11 10:36:26 -05:00
dcbf285fae
Un-ignore message controller tests.
2021-02-11 10:14:48 -05:00
71510a8199
Decommission the old directory cache.
2021-02-10 11:02:11 -05:00
2f105ed0a4
Add support for remote client deprecation
2021-02-09 12:24:44 -05:00
6c0de89de8
Remove scary message-throttling feature flags.
2021-02-08 11:55:01 -05:00
aa99e202b4
Clarify behavioral contract of the pre-key store
2021-02-08 11:45:57 -05:00
04728ea4bc
Drop the old Postgres-based pre-key store.
2021-02-08 11:40:43 -05:00
d09b36b1d5
Fix deprecated imports for Java6Assertions and Matchers
2021-02-08 09:45:33 -06:00
a5dd4f5fac
Remove signaling key from Device and AccountAttributes
2021-02-08 09:45:33 -06:00
3298db8683
Split traffic between the Postgres and Dynamo pre-key stores.
2021-02-04 15:18:53 -05:00
d4d9403829
Add a Dynamo-backed key store.
2021-02-04 15:18:53 -05:00
5e1a572bd8
Track impact of unsealed sender rate limits ( #374 )
2021-02-03 08:58:47 -08:00
0dcb4b645c
Build Dynamo DB backed Message Store ( #358 )
...
* Work in progress...
* Finish first pass draft of MessagesDynamoDb
* Use begins_with everywhere for destination device id
* Remove now unused methods
* First basic test built
* Add another test case
* Remove comment
* Verify more of the message contents
* Ensure all methods are tested
* Integrate MessagesDynamoDb into the MessagesManager
This change plugs the MessagesDynamoDb class into the live serving
flow in MessagesManager.
Tests are not yet as comprehensive for this big a change as they
should be, but they now compile and pass so checkpointing here with a
commit.
* Put DynamoDB before RDBS when deleting specific messages
* Extract method
* Make aws sdk version into a property
* Rename clientBuilder
* Discard messages with no GUID
* Unify batching logic into one function
* Comment on the source of the value in this constant
* Inline method
* Variable name swizzle
* Add timers to all public methods
* Add missing return statements
* Reject messages that are too large with response code 413
* Add configuration to control dynamo DB timeouts
* Set server timestamp from the ReceiptSender
* Change to shorter key names to optimize IOPS
* Fix tests broken by changing column names
* Fix broken copyright template output
* Remove copyright template error text
* Add experiments to control use of dynamo and rds in message storage
* Specify instance profile credentials for the dynamic configuration manager
* Use property for aws sdk version
* Switch dynamo to instance profile credentials
* Add metrics to the batch write loop
* Use placeholders in logging
2021-02-03 10:03:19 -06:00
1a27c7eabc
Add a (failing) test for new Ivory Coast phone numbers.
2021-02-01 21:25:14 -05:00
408b959441
Require a push challenge when registering (or else require a captcha).
2021-02-01 20:44:21 -05:00
35fc98a188
Add an experiment enrollment manager.
2021-02-01 11:08:16 -05:00
92f6a79e1f
Add a dynamic configuration manager
2021-02-01 11:01:58 -05:00
8f94ed68a3
Ignore expired devices when checking for GV1->GV2 migration capability.
2021-01-30 16:55:05 -05:00
92a0deffcf
Add more robust tests for about/emoji fields.
2021-01-27 10:34:13 -05:00
24ea6a9f1d
Revert "Temporarily disable registration abuse system"
...
This reverts commit 22ef058cb6
2021-01-25 14:58:27 -05:00
f10be893ce
Drop the old feature flag controller.
2021-01-25 14:55:57 -05:00
c606c1664f
Add admin tasks for listing, setting, and deleting feature flags.
2021-01-25 14:55:57 -05:00
225932b4c9
Add emojis/"about" text to profiles
2021-01-20 15:42:47 -05:00
6b850b9894
Allow (versioned) profile names up to 380 base64 characters long.
2021-01-20 11:08:10 -05:00
943a5d1036
Shard push scheduling cache
2021-01-19 15:50:12 -05:00
0a23ce870a
Allow message persisters to be disabled by a feature flag.
2021-01-17 11:13:12 -05:00
a57ce1dd17
Add machinery to allow a percentage of message sends to succeed.
2021-01-15 17:05:16 -05:00
81c1ba6eef
Respond to all "message send" attempts with HTTP/503.
2021-01-15 15:34:14 -05:00
9c53d818f4
Use separate clusters for message cache read/write operations.
2021-01-15 13:51:39 -05:00
e5a2c1ab10
Always return an empty list of prekeys.
2021-01-15 12:27:10 -05:00
67ed035b36
Retry serializable key transactions.
2021-01-13 17:38:29 -05:00
2e01da5ec1
Add a task to enable/disable accelerated crawling.
2021-01-11 19:29:18 -05:00
9412a7424c
Return HTTP/429 whenever somebody tries to get contacts from the old directory system.
2021-01-11 19:29:10 -05:00
f8cbb4f386
Temporarily suspend client version metrics to reduce load on our metric aggregator.
2021-01-11 14:04:44 -05:00
86ccaa52a5
Allow configuration of multiple directory account crawler listeners ( #325 )
...
* Allow configuration of multiple directory account crawler listeners
Only one should update the local redis directory. This one is marked
with replicationPrimary true. The others in the list only serve to
issue replication requests over to CDS replication load balancers.
* Update one more metric name
2021-01-10 17:11:02 -06:00
1b53f10091
Reload scripts across the whole cluster if one shard is missing the script.
2021-01-10 15:00:12 -05:00
bac268a21c
Don't send a reply to clients until messages are safely in a non-volatile store.
2021-01-10 13:03:40 -05:00
22ef058cb6
Temporarily disable registration abuse system
2021-01-09 15:57:55 -05:00
9ee6419bc0
Publish directory updates to multiple SQS queues.
2021-01-08 18:07:18 -05:00
3bf0188e7f
Turn off alphanumeric sender ID for all countries.
2021-01-08 06:18:53 -05:00
91fc0fd623
Revert "Delete data in the storage service when deleting accounts."
...
This reverts commit ff1a721d5b
2021-01-08 06:18:39 -05:00
a4d0c17efd
Record OS versions for iOS requests.
2020-12-23 11:36:31 -05:00
ff1a721d5b
Delete data in the storage service when deleting accounts.
2020-12-23 11:35:38 -05:00
c870a1bbd5
Introduce a storage service client.
2020-12-23 11:35:38 -05:00
e551fd2c1b
Revert "Pause checks for GV1 migration when checking for capability downgrades."
...
This reverts commit e7745db36e
2020-12-10 17:02:41 -05:00
34a11c2338
Record OS versions for desktop and SDK versions for Android.
2020-12-10 17:02:05 -05:00
e524ff965d
Add a utility method for getting client platform tags from UA strings for metrics.
2020-12-10 17:01:46 -05:00
92fde83b3a
Discard oversized messages bound for desktop clients via websockets.
2020-12-07 15:03:35 -05:00
299b680013
Always include UUID in UD certificate ( #300 )
2020-12-01 08:56:55 -06:00
00a3e562dc
Force use of UCS-2 instead of GSM-7 for SMS to China ( #297 )
2020-11-20 14:41:48 -06:00
e7745db36e
Pause checks for GV1 migration when checking for capability downgrades.
2020-11-17 09:25:12 -05:00
474b879b16
Only notify CDS if an account attribute change actually changes an account's discoverability.
2020-11-16 10:54:12 -05:00
325e65db7f
Expand UA parsing tests to cover OS details in desktop strings.
2020-11-11 13:05:18 -05:00
604287244f
Update copyright statement on all source files
...
IntelliJ Copyright Profile used to automate this.
2020-11-04 11:55:35 -05:00
5587b7d469
Expose gv1-migration on profile endpoint
2020-10-28 13:00:57 -04:00
26870d134f
Set source UUID when delivering envelopes from message cache/db on websocket
2020-10-28 12:38:32 -04:00
c2db2d3cbd
Add GV1 Migration capability
2020-10-27 16:17:21 -04:00
05d9ec673e
Send push notifications if websockets close before all messages are delivered
2020-10-27 16:02:55 -04:00
96d3a69479
Use container-managed executors for APN/GCM senders.
2020-10-23 11:30:03 -04:00
23ca011ac1
Record account deletion reasons.
2020-10-20 19:00:34 -04:00
99f488d48f
Drop websocket connection names (unused for a while now).
2020-10-19 11:24:35 -04:00
05929871c9
Rename PushSender to MessageSender and add docs.
2020-10-19 11:24:35 -04:00
74b3daa70a
Collapse WebsocketSender into PushSender.
2020-10-19 11:24:35 -04:00
3a84775912
Log cluster topology change events, too.
2020-10-13 16:07:08 -04:00
a9d0aa136d
Add OS-reported metrics for cached/buffered memory.
2020-10-11 13:43:15 -04:00
39c09733d3
Add /v1/payments/auth endpoint
2020-10-08 10:51:01 -04:00
e1c397993d
Require Android clients to support the gv2-3 capability
2020-10-06 16:49:49 -04:00
2b6811cb1b
Really delete old accounts instead of just removing their push channels.
2020-10-06 11:04:47 -04:00
c82496b972
Remove the "repair queue metadata" script.
2020-10-05 16:57:16 -04:00
bb087caddc
Don't panic if a queue exists, but is empty when repairing metadata.
2020-10-04 16:09:56 -04:00
5e3f8b9c2e
Disallow insertion of duplicate messages.
2020-10-04 15:34:14 -04:00
1ccfe928f7
Add a test to make sure that we don't double-insert messages with the same GUID.
2020-10-04 15:34:14 -04:00
952cfae4e6
Repair queue metadata before persisting queues.
2020-10-04 15:25:06 -04:00
02a2c3224f
Discard unused feature flag constants/mocking.
2020-10-04 10:48:42 -04:00
fc05529574
Let MessagePersister manage its own worker thread.
2020-10-03 11:43:34 -04:00
76665dd56e
Retry Redis commands that time out.
2020-10-01 17:17:07 -04:00
885fa6beae
Add tests for Device#isEnabled.
2020-10-01 12:54:35 -04:00
65cdd5fcbe
Drop the 365-day check when deciding if an account is enabled.
2020-10-01 12:54:35 -04:00
0c6f05f34a
Add a (failing!) test for sending a sealed-sender message after a non-sealed-sender message.
2020-10-01 11:06:43 -04:00
57d2ef8740
Return queues to the "to persist" list if something goes wrong during persistence.
2020-09-30 10:39:56 -04:00
a97e0982e3
Add an integration test for message persistence.
2020-09-30 10:39:56 -04:00
eaa2060d84
Fix an incorrect locking key and some previously-suppressed lock contention issues.
2020-09-30 10:39:56 -04:00
c7230ccbb0
Remove messages from the cache in bulk.
2020-09-29 10:58:02 -04:00
fc71ced660
Persist messages in batches.
2020-09-29 10:58:02 -04:00
599cd766e1
Let Dropwizard manage persister thread lifecycles.
2020-09-29 10:58:02 -04:00
e64c8007c0
Detect GV2 capability in non-gcm Android devices
2020-09-28 15:54:10 -04:00
e6d4620af1
Only allow linking desktop clients if they support the third-generation GV2 capability.
2020-09-25 17:08:32 -04:00
656e6db846
Only consider desktop devices GV2-capable if they send the third-gen GV2 capability.
2020-09-25 17:08:32 -04:00
30474e3a2b
Add a test for message ordering.
2020-09-25 11:41:58 -04:00
a553eba574
Add an API endpoint for deleting accounts.
2020-09-25 11:39:17 -04:00
61f515670c
Add plumbing for deleting accounts and all associated data.
2020-09-25 11:39:17 -04:00
789af0f8a6
Add support for deleting keys associated with an account.
2020-09-25 11:39:17 -04:00
86fae58c96
Add support for deleting account entities from the database.
2020-09-25 11:39:17 -04:00
66a04ed730
Don't explicitly notify clients when messages get persisted.
2020-09-23 14:51:02 -04:00
7e14a0bc30
Drop pub/sub operations from WebsocketConnection.
2020-09-23 14:51:02 -04:00
77de0f86dc
Require desktop clients to send the new gv2-2 capability flag.
2020-09-23 12:05:58 -04:00
e9e18afb4a
Add a (failing) integration test demonstrating an infinite loop.
2020-09-22 10:21:33 -04:00
62c31eb202
Revert "Revert keyspace delivery for all messages"
...
This reverts commit 4dc49604b6
2020-09-22 10:21:33 -04:00
5986145282
Add a second-generation GV2 capability and ignore the old capability for iOS devices.
2020-09-21 18:57:53 -04:00
83f9eacac4
Refactor UserAgentTagUtil to parse UA strings with UserAgentUtil.
2020-09-21 12:24:08 -04:00
baab6b951b
Add a general utility class for parsing user-agent strings.
2020-09-21 12:24:08 -04:00
ebc3a251b7
Drop the UUID addressing capability flag entirely.
2020-09-14 15:36:29 -04:00
4dc49604b6
Revert keyspace delivery for all messages
...
* Revert "Send all messages via keyspace notifications when a feature flag is enabled."
This reverts commit fadcf62166c02b25576689788fa665a052e2ee8f158e5004b76f9ff3be37f766c577438f53152c3e7bbc88d71668256d2343
2020-09-14 15:35:10 -04:00
fadcf62166
Send all messages via keyspace notifications when a feature flag is enabled.
2020-09-11 13:12:17 -04:00
a052e2ee8f
Refactor: collapse state into semaphores/atomic booleans.
2020-09-11 13:12:17 -04:00
6f9ff3be37
Avoid querying the database if we think all new messages are in the cache.
2020-09-11 13:12:17 -04:00
f766c57743
Query for more stored messages if an update happens while we're already processing a batch.
2020-09-11 13:12:17 -04:00
8f53152c3e
Only send the "queue cleared" message once per websocket session.
2020-09-11 13:12:17 -04:00
7bbc88d716
Let processStoredMessages handle requery logic.
2020-09-11 13:12:17 -04:00
68256d2343
Only allow one thread to process stored messages at a time.
2020-09-11 13:12:17 -04:00
f88c440c48
Automatically retry when Twilio returns unreachable ( #190 )
...
* Parse and log the Twilio error code
* Automatically retry without sender ID when Twilio returns unreachable
* Remove attempt count and pass around whether or not sender id was used
2020-09-10 13:58:39 -05:00
2c6b646d87
Enforce no capability downgrade on device verification
2020-09-09 16:05:00 -04:00
e7572094b5
Require all enabled devices to support GV2.
2020-09-09 16:05:00 -04:00
fdef21a871
Record and listen for ephemeral messages in a separate queue.
2020-09-09 14:42:09 -04:00
d40cff8a99
Revert "Add a system for storing, retrieving, and notifying listeners about ephemeral (online) messages."
...
This reverts commit 06754d6158
2020-09-08 15:55:09 -04:00
06754d6158
Add a system for storing, retrieving, and notifying listeners about ephemeral (online) messages.
2020-09-08 11:14:42 -04:00
1d5087374e
Jettison UUID-or-E164 plumbing in favor of UUID-only.
2020-09-08 09:30:47 -04:00
8356264fe0
Rename RedisClusterMessagesCache and related classes to just MessagesCache.
2020-09-08 09:30:47 -04:00
18ecd748dd
Entirely discard the old message cache machinery.
2020-09-08 09:30:47 -04:00
5c04f2634a
Use a dedicated executor service for dispatching keyspace notifications.
2020-09-03 11:04:48 -04:00
81e8143a43
Rely solely on the clustered message cache.
2020-09-02 11:57:33 -04:00
8409986ef5
Mirror persistence operations from the new persister to the old persister.
2020-09-02 11:02:40 -04:00
2b50367d7f
Put message persisters behind feature flags.
2020-09-02 11:02:40 -04:00
5221828705
Increase maximum sticker size to 300 kibibytes
...
In preparation for animated stickers, allow stickers to be up to 300
kibibytes.
2020-09-01 10:55:26 -04:00
ce026e7ad0
Don't send contacts to CDS if they've opted out of discoverability. (SERVER-130)
2020-08-27 15:58:02 -04:00
58e3122dab
Add a discoverableByPhoneNumber account attribute. (SERVER-129)
2020-08-27 15:58:02 -04:00
32b18c9509
Add an endpoint for getting the current state of feature flags.
2020-08-27 13:18:12 -04:00
acf52ad8a3
Make feature flag manager tests use a real database to avoid over-mocking.
2020-08-27 13:18:12 -04:00
08dd493f98
Don't report exceptions as part of traffic metrics.
2020-08-27 13:17:57 -04:00
07bbe7dfb2
Return to an async model for push notification latency.
2020-08-27 10:51:44 -04:00
0aa1b80e3e
Add a command for persisting any detached messages in the old message cache.
2020-08-27 10:51:12 -04:00
5ac390281e
Add an abstract base class for Redis singleton tests.
2020-08-27 10:51:12 -04:00
ac465c5a18
Add a Lettuce-based Redis singleton client.
2020-08-27 10:51:12 -04:00
1ef3546822
Add support for server-side feature flags
2020-08-26 20:27:33 -04:00
e74ad2b555
Make RedisClusterMessagesCache a Managed class.
2020-08-25 10:58:01 -04:00
56b27ea785
Record experiment outcomes with timers instead of counters.
2020-08-25 10:57:44 -04:00
2d75f59d33
Add support for UUID-only delivery certificates. (SERVER-132)
2020-08-20 17:05:53 -04:00
34bf5112e0
Drop TimeProvider.
2020-08-20 15:40:24 -04:00
bfe18d1d28
Re-nerf the clustered message persister.
2020-08-20 15:38:09 -04:00
6a76afc20d
Add a test to make sure the persister is respecting persist delays.
2020-08-20 15:38:09 -04:00
9c469c2f96
Base persister tests on a real Redis cluster.
2020-08-20 15:38:09 -04:00
af34b43a8d
Reactivate the message notification experiment.
2020-08-19 11:39:05 -04:00
42ea7a9814
Revert Lettuce connection pooling.
2020-08-18 16:21:42 -04:00
d243b73678
Make Lettuce connection pools configurable. Double the default size.
2020-08-18 12:20:12 -04:00
dc28d063aa
Reactivate the explicit client presence experiment.
2020-08-17 11:34:27 -04:00
bb6045c1d0
Disarm the client presence manager experiment.
2020-08-15 20:23:05 -04:00
f1a74b5939
Disarm new message keyspace notifications.
2020-08-15 20:23:05 -04:00
6fb9038af1
Move to a synchronous, pooled connection model for Redis clusters.
2020-08-14 17:15:56 -04:00
27f721a1f5
Update to resilience4j 1.5.0.
2020-08-14 17:15:56 -04:00
5717dc294e
Combine the read/write breakers for Redis clusters.
2020-08-14 17:15:56 -04:00
ae0f8df11b
Break out FaultTolerantPubSubConnection as its own thing so different use cases can have their own subscription space.
2020-08-14 17:15:56 -04:00
77460ba502
Remove keyspace notification configuration checks because AWS doesn't support `CONFIG GET`.
2020-08-13 15:32:25 -04:00
8d3316ccd6
Listen for new messages via keyspace notifications.
2020-08-13 12:17:04 -04:00
2c29f831e8
Add an explicit client presence system.
2020-08-13 10:56:26 -04:00
9457325119
Add pub/sub affordances to FaultTolerantRedisCluster.
2020-08-13 10:56:26 -04:00
189f8afcc9
Warm up the test cluster before running tests to avoid transient startup jitters.
2020-08-13 10:56:26 -04:00
5fad8f74b1
Factor MessagePersister into its own class.
2020-08-11 15:50:22 -04:00
e35e34d2e0
Move operation-mirroring logic to MessagesManager.
2020-08-11 15:50:22 -04:00
31a215d4d6
Use "global." instead of "g." as the prefix for global config options.
2020-08-11 11:55:35 -04:00
b97158bf7b
Create global remote config controllable in the signal server configuration ( #127 )
...
* Add global config controller through file rather than database
* Do no permit attempting to set or delete global config entries
2020-08-10 16:31:15 -05:00
3286c5e174
Disable Redis persistence for tests.
2020-08-06 11:22:51 -04:00
4fa3a136ad
Remove arbitrary SMS and add a NANPA message service ( #123 )
...
* Remove arbitrary SMS code
This code has run its course and is no longer needed for now.
* Add elements to sample config that were left out
* Add a messaging service for NANPA
* Fixup sample config capitalization
2020-08-05 13:35:11 -05:00
57e1339230
Further restrict user agent pattern matching ( #120 )
...
* Further restrict user agent pattern matching
* Add static qualifier to method
2020-08-04 12:58:16 -05:00
0bc5566976
Mirror delete-after-persist operations to the clustered message cache.
2020-08-04 10:58:41 -04:00
ad97731d46
Reduce the maximum number of versions in play to 1,000.
2020-08-03 15:42:15 -04:00
40684a93a2
Restrict user-agent version matching to a more confined space.
2020-08-03 15:42:15 -04:00
f3b644ceb8
Update the push latency manager to use UUIDs and a Redis cluster.
2020-08-03 15:36:02 -04:00
901ba6e87f
Added a push latency manager.
2020-08-03 15:36:02 -04:00
76389bd584
Clear would-be-persisted messages from the cache cluster, but don't store them to the database.
2020-07-30 19:14:39 -04:00
7bf8650d59
Un-manage FaultTolerantRedisCluster so it shuts down at JVM shutdown instead of Jetty shutdown.
2020-07-30 18:37:38 -04:00
baf563e46d
Temporarily disarm the actual persisting part of the message persister.
2020-07-30 17:12:37 -04:00
beac73b6c8
Add a cluster-capable message persister
2020-07-30 11:39:14 -04:00
f9f93c77e2
Use UUIDs instead of phone numbers as account identifiers in clustered message cache
2020-07-30 11:39:14 -04:00
6fc1b4c6c0
Add a cluster-backed message cache.
2020-07-30 11:39:14 -04:00
639898ec07
Expand Experiment to deal with async suppliers and Optionals.
2020-07-30 11:39:14 -04:00
3d3790fdbc
Add binary execution methods to ClusterLuaScript.
2020-07-30 11:39:14 -04:00
69c8968cb0
Add byte-array-based methods to FaultTolerantRedisCluster.
2020-07-30 11:39:14 -04:00
7f8f2641f6
Simplify registration lock counting by avoiding inactive accounts.
2020-07-28 11:48:20 -04:00
022dbb606f
Count registration lock versions when crawling the account database.
2020-07-28 11:48:20 -04:00
eea073f882
Decommission the old cache.
2020-07-28 10:29:28 -04:00
fc1d88f5bb
Read exclusively from the cache cluster.
2020-07-27 15:11:40 -04:00
89bafea61f
Move SMS strings to configuration
2020-07-27 11:23:21 -05:00
0fbf31ec98
Clear each cluster node individually.
2020-07-22 11:12:21 -04:00
db9b7ca447
Fix slot assignment when building a cluster for tests.
2020-07-22 11:04:10 -04:00
eecc71c77f
Revert batch message storage. ( #95 )
2020-07-20 16:28:32 -04:00
215125de26
Update tests.
2020-07-20 10:30:29 -04:00
247d869a5c
De-randomize message tests to minimize flakiness.
2020-07-14 18:46:39 -04:00
b7e0e5a356
Create a strategy class to decide which sender id to use
...
The rules around selecting sender ids can get complicated with some
countries not supporting it and others requiring pre-registration that
may result in having a different sender id for that country than
others. This strategy class handles the logic of dealing with this
expanded configuration and applying the appropriate sender id or none
when it's not appropriate to do so at all.
2020-07-14 10:53:48 -05:00
c5d0d4acd0
Revert "Move rate limiter logic to Lua scripts"
...
This reverts commit b585c6676d
2020-07-09 12:30:25 -04:00
dcb11f7606
Log errors from experiments.
2020-07-07 15:17:25 -04:00
933ce42d5a
Test rate limiters against a real cluster.
2020-07-07 15:17:25 -04:00
e021286eee
Add configuration by country for sending from alpha IDs
2020-07-07 10:17:40 -05:00
0ee7a66033
Keep trying ports until you get one lower than 55535 ( #83 )
...
* Keep trying ports until you get one lower than 55535
* Rename method and change to do...while
* Limit attempts to 11,000 to find an open redis cluster port
2020-07-07 10:12:31 -05:00
42c797ee97
Set the default log level for tests to WARN.
2020-07-07 11:05:39 -04:00
b585c6676d
Move rate limiter logic to Lua scripts
2020-07-06 10:10:13 -04:00
f5ddb0f1f8
Test ClusterLuaScript against a real Redis cluster.
2020-07-02 18:58:30 -04:00
25a5a8db68
Set avatar to null on Account when request is false ( #78 )
2020-06-29 15:53:31 -05:00
a68d91b54c
Resolve some test flakiness by adding a deterministic "wait" mechanism. (SERVER-86)
2020-06-29 12:24:25 -04:00
bbf5e1fa78
Use the UA string from websocket upgrade requests if available.
2020-06-17 15:40:18 -04:00
6adcebb247
Return to just using counters instead of timers for measuring experiment outcomes.
2020-06-17 15:34:02 -04:00
a0bebca1e6
Extend Experiment to report more detail when results don't match.
2020-06-16 16:46:41 -04:00
c17cc07b73
Instrument BlockingThreadPoolExecutor.
2020-06-12 14:43:50 -04:00
9afc433db4
Record exceptions associated with server responses.
2020-06-11 22:08:07 -04:00
f701e3d834
Record distributions of timer values; stop recording error causes.
2020-06-11 11:50:36 -04:00
0671f05c05
Introduce experiment comparison methods for suppliers.
2020-06-11 11:50:36 -04:00
0713da7393
Record experiment results with a timer instead of a counter.
2020-06-11 11:50:36 -04:00
28c765bd9a
Add an in-app-context test for websocket metrics.
2020-06-09 15:45:32 -04:00
ec858b2d4c
Set a timeout for Redis cluster operations and shut down the cluster as part of service shutdown
2020-06-07 18:27:57 -04:00
52310b5dd9
Compare results of reads from old and new Redis caches.
2020-06-07 18:27:11 -04:00
c2a4a2778e
Introduce the Experiment class to compare results from parallel systems.
2020-06-07 18:27:11 -04:00
1388103919
Mirror writes to the cache cluster.
2020-06-06 20:37:48 -04:00
fe1054d58a
Introduce a Lettuce-based fault-tolerant Redis cluster accessor.
2020-06-06 20:37:48 -04:00
ba6ac778fc
Update to Pushy v0.14.1.
2020-06-05 12:21:56 -04:00
228ffcbfce
Differentiate between websocket and "boring" HTTP traffic.
2020-05-28 12:52:49 -04:00
06c82ee87d
Celebrate the diversity of UA strings when generating tags for metrics.
2020-05-27 19:35:42 -04:00
9ba5ee8043
Move UA tag extraction into its own utility class.
2020-05-27 19:35:42 -04:00
eede4e50ca
Use hashed UUID to spread last seen updates over a full day ( #40 )
2020-05-26 13:38:52 -07:00
aa10f63d9f
Add the timestamp using the `add` method.
2020-05-22 17:39:25 -04:00
a13c44d81a
Capture request-level metrics (path, status, client platform/version).
2020-05-20 17:48:19 -04:00
7da9e88c0b
Add hashKey to RemoteConfig
...
This allows the percentages for different entries in remote config to
be aligned so one remote config can be a subset of another.
2020-05-13 11:08:22 -07:00
1c73c91133
Report the number of days until the CDS CA cert expires as a metric so we can set an alarm.
2020-05-12 12:57:11 -04:00
001a9310c3
Support device transfers (SERVER-41, SERVER-42) ( #32 )
...
This change introduces a `transfer` device capability and account creation argument in support of the iOS device transfer effort.
2020-05-12 12:23:18 -04:00
acfbab5915
Update to Pushy v0.13.11.
2020-05-04 17:50:35 -04:00
48c324fe86
Use a static sequence of randomness in tests
...
The RemoteConfigControllerTest#testMath unit test would occassionally
fail because randomness doesn't necessarily group into expected ranges
over a finite trial count. This changes the test to use a predefined
PRNG sequence instead of one that varies with each test so that the
test will no long randomly fail.
2020-04-29 17:31:43 -07:00
0c495e7e72
Workaround lack of internal retry on transaction rollback
...
The get endpoint for key fetching can fail if the transaction cannot
complete because of simultaneous modification. Clients currently
receive 500 from this and retry if it happens, but this test case runs
into it without retrying and then complains that not all the threads
completed successfully. This workaround adds some retry attempts.
2020-04-29 17:10:13 -07:00
50ccfee201
Allow remote config to send non-boolean values
...
This version of remote config allows non-boolean values to be returned
to clients but unfortunately limits the configuration to only one
value or another. There is no way to configure more than two values
for the same key with this setup.
2020-04-29 10:51:10 -07:00
95f0ce1816
Support for advertising payment addresses on profile
2020-04-22 12:32:53 -07:00
1408ac77f9
Make storageCapable a boolean result rather than an auth token
2020-04-09 10:19:49 -07:00
7e97d10ae1
Fix account dropping new style registration locks
2020-04-06 09:27:23 -07:00
41286650cc
Create attachments V3 endpoint for CDN2 on GCP
...
In preparation for resumable uploads, this creates a separate
attachment authorization endpoint that creates a signed URL for
accessing GCP Storage through Signal's CDN2. This should allow Signal
clients to do byte-level resume of media uploads.
2020-04-02 10:20:42 -07:00
3c8e7c6c10
Add storage capability and return KBS creds on rereg w/ storage set
2020-03-27 10:45:48 -07:00
ac1153c7cf
Additional limits
2020-03-14 18:10:07 -07:00
3b1672a4a7
Update zkgroups to 0.4.0
2020-03-14 16:30:13 -07:00
e5746c19cf
Support for GV2 capability flag
2020-02-07 11:53:28 -08:00
e399f9e851
Generate external creds for KBS based on UUID
2020-01-22 13:47:33 -08:00
e4e20c2d25
Add support for UUID buckets in remote config
2020-01-22 11:28:08 -08:00
08a70664f4
Support for getting/setting remote config variables
2020-01-21 13:38:58 -08:00
75fc35ee4b
Parameterize access to zk operations
2020-01-21 11:29:08 -08:00
ba3102d667
Support for versioned profiles
...
Includes support for issuing zkgroup auth credentials
2020-01-21 11:04:06 -08:00
8a9fed64f2
Support for first/last profile name length
2020-01-13 18:55:04 -08:00
71c7e30548
Increase max size for sticker manifest
2019-12-19 10:29:47 -08:00
886db1a2c3
Bump max sticker count to 201
2019-12-18 17:08:51 -08:00
b4c06db031
Make redis failures on write-back retrieve non-fatal
2019-11-20 12:36:22 -08:00
2b987e6e93
Usernames can't start with numbers
2019-11-20 12:36:22 -08:00
523134f24b
Username reservation table
2019-11-20 12:36:22 -08:00
99c228dd6d
Support for setting and looking up usernames
2019-11-20 12:36:22 -08:00
c623f70caa
Add support for capabilities
2019-11-14 13:36:40 -08:00
f16b783378
return backup, not storage, credentials for reg lock
2019-11-05 10:36:33 -08:00
be4b75932b
since onCrawlChunk() is now protected, we need to invoke timeAndProcessChunk() in our unit tests
2019-10-29 18:20:03 -07:00
eddfacd0f4
add timers to the account crawler listeners
2019-10-25 21:30:48 -07:00
69742839c0
uuid-based account crawler
2019-08-27 14:42:14 -07:00
20b5f0e681
Reset cache index
2019-08-27 14:08:50 -07:00
3803b8f284
Fix for jedis pool deadlock
...
1) Remove nested pool checkouts
2) Add a max wait so it won't block forever on deadlock
2019-08-27 14:02:42 -07:00
cf78047830
revert to phone number-based account crawler
2019-08-26 14:00:15 -07:00
79f2efdfd9
Make UUID in sealed sender certificate optional for buggy clients
2019-08-26 11:09:54 -07:00
07822b371f
replicate uuids to contact discovery
2019-08-26 11:09:54 -07:00
7a3a385569
Support for UUID based addressing
2019-08-26 11:09:54 -07:00
10724fee04
Support for sticker pack uploads
2019-07-24 16:29:56 -07:00
11902dec3c
Support for v2 registration lock
2019-07-11 18:15:14 -07:00
4fdbe9b9ff
Support for push preauth
2019-07-11 18:15:10 -07:00
a6e7e30177
Add requester to recaptcha validation
2019-07-11 17:31:34 -07:00
5b69ff7e94
Break out keys database and accounts database
2019-06-19 17:16:37 -07:00
bc0c6be4c5
We don't need to support disabled accounts for the signed PK API
2019-06-12 12:32:15 -07:00
105a38a7db
Update gcm-sender-async to use jdk11 httpclient
2019-05-30 10:46:40 -07:00
a029768d24
Reenable account cleaner
2019-05-10 10:42:42 -07:00
4d9c9206cf
Delay processing FCM uninstalled feedback
...
Check to make sure client is not still active before unregistering,
since FCM feedback seems to be often erroneous
2019-05-07 10:04:22 -07:00
35116f9229
Clean up concepts of enabled account state
...
1) Rename "active" methods to be "enabled," since they aren't
really about "activity."
2) Make authentication fail if a device or account is in dissabled
state.
3) Let some controllers authenticate accounts that are in a
disabled state.
2019-05-04 12:31:50 -07:00
a1f90cd39b
Temporarily disable account cleaner
2019-05-03 12:09:01 -07:00
45dc7459b8
Temporarily disable GCM unregistered feedback
2019-05-03 11:51:21 -07:00
6877b663f1
enable up to 40 account updates per chunk in AccountCleaner
2019-05-03 10:58:57 -07:00
3c69f81a10
expire accounts explicitly
2019-05-02 21:14:57 -07:00
d316d57e5d
fix DirectoryController tests
2019-05-02 19:20:23 -07:00
0c81556b90
Switch websocket-resources from ListenableFuture to CompletableFuture
2019-05-02 15:05:44 -07:00
d0d375aeb7
Break out into a multi-module project
2019-05-01 13:19:11 -07:00
Chris Eager