ad6c271f9d
Add dynamic configuration for signup captcha by country code
2021-03-30 10:22:02 -05:00
c10d9603ad
Add metric for international PreKey requests
2021-03-29 14:52:04 -05:00
91bd061110
Migrate deprecated Lettuce method and enum usages
2021-03-29 14:51:26 -05:00
83aa59f4dd
Align messages_cluster client name
2021-03-29 14:51:26 -05:00
3745a0b81d
Update from 684 to 776 for payment address length to account for signature
2021-03-27 00:45:47 -05:00
7e29ed1cc7
Block attempts to set wallet addresses from unsupported countries.
2021-03-26 12:21:11 -04:00
5965f0fd22
Add a dynamically-configured list of allowed country codes for payments.
2021-03-26 12:21:11 -04:00
a816aa0186
Revert "Add a storage client method for checking wheter a user has a stored manifest."
...
This reverts commit 8b6012f8a8
2021-03-22 10:32:38 -04:00
a7bad20eae
Revert "Add an admin command for printing a subset of account data."
...
This reverts commit a288b9df8e
2021-03-22 10:32:38 -04:00
089b6b1644
Retry attempts to get messages after a delay; close connections after a finite number of retries.
2021-03-22 10:32:25 -04:00
7509520883
Make sure to release the semaphore even if something goes wrong getting messages.
2021-03-22 10:32:25 -04:00
e5ae0572c5
Add android-2021-03 verification format
2021-03-18 12:49:50 -05:00
63dac3bd9f
Migrate AccountControllerTest to JUnit 5
2021-03-18 12:45:20 -05:00
19295eef46
Add dimensional metrics for account creation and verification
2021-03-18 12:45:20 -05:00
0bc1369e04
Work through the full list of supported locales when choosing a language for voice verification.
2021-03-18 13:29:09 -04:00
ca2f7d2eed
Parse locale strings when sending voice verification codes.
2021-03-18 13:29:09 -04:00
3ea535a412
Lower logging level for common failures and record failure metrics.
2021-03-18 12:43:31 -04:00
a288b9df8e
Add an admin command for printing a subset of account data.
2021-03-18 12:43:12 -04:00
8b6012f8a8
Add a storage client method for checking wheter a user has a stored manifest.
2021-03-18 12:43:12 -04:00
ad838b4827
Add isEnrolled(e164, experiment) for pre-registration experiments
2021-03-16 13:09:10 -05:00
25f603efc9
Add DynamicPreRegistrationExperimentEnrollmentConfiguration
2021-03-16 13:09:10 -05:00
152c927929
Reformat to match EditorConig
2021-03-16 13:09:10 -05:00
b5bd16c6a9
Migrate DynamicConfigurationTest to JUnit 5
2021-03-16 13:09:10 -05:00
14bfa83bb8
Migrate ExperimentEnrollmentManagerTest to JUnit 5
2021-03-16 13:09:10 -05:00
5dc8086968
Migrate Twilio numbers from static to dynamic configuration
2021-03-16 12:53:51 -05:00
efe7f2e4c1
Remove vestiges of per-country sender ID logic/configuration.
2021-03-10 14:58:46 -05:00
fb2fc2335a
Require messaging service IDs; remove fallback-to-random-number logic.
2021-03-10 14:58:46 -05:00
345e116699
Place our trust in a Twilio message service.
2021-03-10 14:58:46 -05:00
e50a1c0646
Revert "Turn off alphanumeric sender ID for all countries."
...
This reverts commit 3bf0188e7f
2021-03-10 14:58:46 -05:00
3cdc58200a
Copy headers from the initial websocket upgrade request into subsequent resource requests.
2021-03-10 14:54:11 -05:00
933dd81d82
Allow callers to specify a TLS version when constructing a FaultTolerantHttpClient.
2021-03-10 10:53:33 -05:00
a1434524a4
Allow the storage service client to trust the Signal CA root.
2021-03-10 10:53:33 -05:00
cdc6afefe2
Add an affordance for providing a custom trust store to FaultTolerantHttpClient.
2021-03-10 10:53:33 -05:00
738ec2a38e
Use ForwardedIpUtil everywhere we're handling X-Forwarded-For values.
2021-03-10 10:37:10 -05:00
07886a9722
Introduce a utility class for working with forwarding chains in HTTP headers.
2021-03-10 10:37:10 -05:00
fde1b49729
Introduce a set of gauges for various network statistics as reported by `nstat`.
2021-03-09 11:45:53 -05:00
58210141f4
Discard unhelpful message count histograms.
2021-03-08 16:57:26 -05:00
af2a8548c3
Use Durations everywhere, drop unused constructors, and add tests.
2021-03-05 12:47:36 -05:00
1faedd3870
Return Retry-After time to clients when they are rate limited ( #421 )
...
* Return Retry-After time to clients when they are rate limited
* Update based on feedback
- New exception type that is mapped differently
- Always report time until allowed on rate limits
- Consume and transform into a differnt exception if we think it will be
allowed later
2021-03-05 10:23:03 -07:00
f57a4171ba
Gather IP-based metrics for international, unsealed-sender messages.
2021-03-05 11:54:01 -05:00
df9dc82de5
Record days since last seen when somebody's "last seen" date changes.
2021-03-01 15:31:53 -05:00
eb6fe11da1
Add tools to decline messages from senders meeting specific conditions
2021-02-24 18:09:26 -05:00
0ee3f0a5b5
Fix a goof where the international unsealed sender country meter was incrementing unconditionally.
2021-02-23 16:45:47 -05:00
7dabc92447
Actually increment the international unsealed-sender counter instead of just declaring its existence.
2021-02-23 15:36:36 -05:00
5354104128
Only apply unsealed sender rate limits to targeted country codes.
2021-02-23 14:17:16 -05:00
a5118e4daa
Record push challenge presence/outcomes by country.
2021-02-23 13:36:37 -05:00
b5ade5dc12
Only return payment address from latest profile ( #408 )
...
* Only return payment address from latest profile
* Rename `currentVersionedProfile` to `currentProfileVersion`
* Change return type to Optional
* Update service/src/main/java/org/whispersystems/textsecuregcm/controllers/ProfileController.java
Co-authored-by: Jon Chambers <63609320+jon-signal@users.noreply.github.com>
* Fix broken test
Co-authored-by: Jon Chambers <63609320+jon-signal@users.noreply.github.com>
2021-02-23 12:29:47 -06:00
fff8c72f42
Record the rate of unsealed-sender messages from accounts with a non-push-capable master device.
2021-02-23 12:26:23 -05:00
06ca5f14fc
Record the age of accounts that send unsealed-sender messages.
2021-02-23 12:16:45 -05:00
8c9d871268
Log different messages depending on whether rate limits are actually enforced.
2021-02-23 12:16:45 -05:00
7266eeee7a
Record the rate of international unsealed-sender attempts.
2021-02-23 12:01:09 -05:00
f85c6bf828
Demonstrate JUnit 5 works by migrating an existing test
2021-02-19 13:30:44 -06:00
93f845610d
Remove payments list from Account
2021-02-19 13:18:30 -06:00
aa8525385a
Temporarily support `IncomingMessage.online` ( #404 )
...
iOS versions prior to 5.5.0.7 send `online` on `IncomingMessage`, rather
than on the top-level entity. This adds a temporary server-side adaptation,
to prevent client-side issues, like persistent typing indicators.
2021-02-18 12:21:16 -06:00
ec783133c1
Close websockets if anything seems fishy at init time; register close handlers early.
2021-02-18 13:20:19 -05:00
71f0aab2c6
Actually start the currency managed lifecycle
2021-02-17 14:28:08 -08:00
6142998b87
Use stripToNull on payments address
2021-02-16 16:12:50 -06:00
142376f360
Add payment address to the versioned profile
2021-02-16 16:12:50 -06:00
ae329e735f
Make quote timestamp based on ftx timestamp
2021-02-16 12:02:39 -08:00
2dbab70c8c
Create utility endpoint for currency conversion
2021-02-16 11:57:34 -08:00
635f669a32
Count slow queue drain events by platform.
2021-02-12 15:43:50 -05:00
5f49772ca6
Control enforcement of unsealed sender rate limits via dynamic configuration.
2021-02-12 15:43:05 -05:00
6332552346
Record the sender's country when we reject unsealed-sender messages.
2021-02-12 15:23:52 -05:00
4fb7afcf7b
Drop the feature flags table.
2021-02-12 15:23:34 -05:00
ff448950ed
Collapse the feature flag system into the dynamic config system.
2021-02-12 15:23:34 -05:00
d6319aeb92
just set OWA,OWP,OWI user agents upon token deletion
2021-02-12 15:22:50 -05:00
8fc6f9c442
when a user unregisters, record his user agent for posterity
2021-02-12 15:22:50 -05:00
e9ea79cc8e
Shorten eviction time to 7 days (to match message retention time).
2021-02-11 14:01:11 -05:00
ad32555cc9
Drop the old push scheduler Redis singleton.
2021-02-11 14:01:11 -05:00
be8a1acca9
Remove message database from the codebase ( #395 )
...
* Remove message database from the codebase
* Remove unused ExperimentEnrollmentManager in test
* Be more stylish
2021-02-11 10:50:03 -06:00
e0ed8fa0b8
Introduce a hyper-log-log-based cardinality rate limiter
2021-02-11 10:36:26 -05:00
dcbf285fae
Un-ignore message controller tests.
2021-02-11 10:14:48 -05:00
ceda459942
Substitute some !Optional.isPresent with Optional.isEmpty
2021-02-10 13:21:44 -06:00
28fe44aea4
Remove messages dynamo db experiment
2021-02-10 10:21:49 -06:00
71510a8199
Decommission the old directory cache.
2021-02-10 11:02:11 -05:00
9cd121c8f6
Record initial queue drain times faceted by client platform.
2021-02-09 19:21:46 -05:00
2f105ed0a4
Add support for remote client deprecation
2021-02-09 12:24:44 -05:00
704d54dd01
Record a distribution of the number of keys available when checked.
2021-02-08 16:50:27 -05:00
bee9b61831
Record a distribution of initial queue depths broken down by platform.
2021-02-08 16:06:21 -05:00
9c6ce08db0
Break down message send attempts by client platform, message ephemerality, and sealed sender status.
2021-02-08 16:06:04 -05:00
6c0de89de8
Remove scary message-throttling feature flags.
2021-02-08 11:55:01 -05:00
aa99e202b4
Clarify behavioral contract of the pre-key store
2021-02-08 11:45:57 -05:00
04728ea4bc
Drop the old Postgres-based pre-key store.
2021-02-08 11:40:43 -05:00
6865cdfce3
Replace all org.hibernate validators with modern javax.validation
2021-02-08 09:45:33 -06:00
d09b36b1d5
Fix deprecated imports for Java6Assertions and Matchers
2021-02-08 09:45:33 -06:00
a5dd4f5fac
Remove signaling key from Device and AccountAttributes
2021-02-08 09:45:33 -06:00
9936b2967e
Don't use signaling key anymore
...
Signaling key was deprecated over 2 years ago. It's time for it to go.
2021-02-08 09:45:33 -06:00
0971613ac0
Remove messages table from account DB
...
This is quite out of date. Messages has been a separate database for a
long long time.
2021-02-08 09:45:33 -06:00
f7d855c59e
Drop unsealed sender rate limit logging from `info` to `debug`.
2021-02-05 11:43:37 -05:00
2fe743649d
Extract common UUID operations into a utility class.
2021-02-05 10:40:27 -05:00
a015237fd2
Don't request data from DynamoDB if we already have it locally.
2021-02-05 10:40:27 -05:00
e1f4deaacc
Add a generalized countItemsMatchingQuery method for DynamoDB stores.
2021-02-05 10:40:27 -05:00
1dceee3fa0
Make DynamoDBConfiguration more consistent.
2021-02-05 10:40:27 -05:00
3a17a7c98f
Explicitly make the dynamic config worker a daemon thread.
2021-02-04 19:05:51 -05:00
3298db8683
Split traffic between the Postgres and Dynamo pre-key stores.
2021-02-04 15:18:53 -05:00
d4d9403829
Add a Dynamo-backed key store.
2021-02-04 15:18:53 -05:00
426e6923ac
Continue to poll for config changes after startup.
2021-02-04 13:46:37 -05:00
b413f665d8
Make DynamicConfigurationManager not-really-Managed.
2021-02-04 11:33:33 -05:00
5e1a572bd8
Track impact of unsealed sender rate limits ( #374 )
2021-02-03 08:58:47 -08:00
0dcb4b645c
Build Dynamo DB backed Message Store ( #358 )
...
* Work in progress...
* Finish first pass draft of MessagesDynamoDb
* Use begins_with everywhere for destination device id
* Remove now unused methods
* First basic test built
* Add another test case
* Remove comment
* Verify more of the message contents
* Ensure all methods are tested
* Integrate MessagesDynamoDb into the MessagesManager
This change plugs the MessagesDynamoDb class into the live serving
flow in MessagesManager.
Tests are not yet as comprehensive for this big a change as they
should be, but they now compile and pass so checkpointing here with a
commit.
* Put DynamoDB before RDBS when deleting specific messages
* Extract method
* Make aws sdk version into a property
* Rename clientBuilder
* Discard messages with no GUID
* Unify batching logic into one function
* Comment on the source of the value in this constant
* Inline method
* Variable name swizzle
* Add timers to all public methods
* Add missing return statements
* Reject messages that are too large with response code 413
* Add configuration to control dynamo DB timeouts
* Set server timestamp from the ReceiptSender
* Change to shorter key names to optimize IOPS
* Fix tests broken by changing column names
* Fix broken copyright template output
* Remove copyright template error text
* Add experiments to control use of dynamo and rds in message storage
* Specify instance profile credentials for the dynamic configuration manager
* Use property for aws sdk version
* Switch dynamo to instance profile credentials
* Add metrics to the batch write loop
* Use placeholders in logging
2021-02-03 10:03:19 -06:00
1a27c7eabc
Add a (failing) test for new Ivory Coast phone numbers.
2021-02-01 21:25:14 -05:00
408b959441
Require a push challenge when registering (or else require a captcha).
2021-02-01 20:44:21 -05:00
35fc98a188
Add an experiment enrollment manager.
2021-02-01 11:08:16 -05:00
92f6a79e1f
Add a dynamic configuration manager
2021-02-01 11:01:58 -05:00
8f94ed68a3
Ignore expired devices when checking for GV1->GV2 migration capability.
2021-01-30 16:55:05 -05:00
ce1a4b94cb
Actually store emoji/about text in the database.
2021-01-27 10:34:13 -05:00
92a0deffcf
Add more robust tests for about/emoji fields.
2021-01-27 10:34:13 -05:00
97b6f6028b
Fix a minor typo in the help text for a feature flag task.
2021-01-25 18:03:38 -05:00
611e8c39ee
Actually drop feature flag config.
2021-01-25 15:20:06 -05:00
01f1c263a6
Add a meter for captcha requests.
2021-01-25 14:58:27 -05:00
24ea6a9f1d
Revert "Temporarily disable registration abuse system"
...
This reverts commit 22ef058cb6
2021-01-25 14:58:27 -05:00
46c800b8b7
Smoosh request logging tasks together rather than having one task for each direction.
2021-01-25 14:58:15 -05:00
f10be893ce
Drop the old feature flag controller.
2021-01-25 14:55:57 -05:00
c606c1664f
Add admin tasks for listing, setting, and deleting feature flags.
2021-01-25 14:55:57 -05:00
225932b4c9
Add emojis/"about" text to profiles
2021-01-20 15:42:47 -05:00
6b850b9894
Allow (versioned) profile names up to 380 base64 characters long.
2021-01-20 11:08:10 -05:00
943a5d1036
Shard push scheduling cache
2021-01-19 15:50:12 -05:00
b25da8ceaa
Don't attempt SMS to iran ( #355 )
2021-01-19 09:13:37 -08:00
10cdb7387d
Be consistent with use of DataSize class
2021-01-18 17:01:43 -06:00
dd436dd1dd
Create a Meter for tracking messages larger than 256kib
2021-01-18 17:01:43 -06:00
13b84635b5
Drop an unused message database index. ( #352 )
2021-01-18 10:26:03 -06:00
27534d408f
Log when messages cache detects topology change ( #354 )
...
Co-authored-by: Moxie Marlinspike <moxie+github@signal.org>
2021-01-17 17:13:23 -08:00
0a23ce870a
Allow message persisters to be disabled by a feature flag.
2021-01-17 11:13:12 -05:00
c355ef8d53
Reduce the message cache thread pool size.
2021-01-16 11:15:25 -05:00
1feb23ba99
Stop periodic topology refreshes.
2021-01-16 03:35:36 -05:00
59a0fd0799
Embiggen message cluster thread pool.
2021-01-16 02:57:04 -05:00
00b5cfcf17
Allow the client presence manager to use an entirely separate cluster.
2021-01-16 02:57:04 -05:00
9e342f253d
Use the same client for inserts and reads in the message cache cluster.
2021-01-16 01:50:40 -05:00
20c48b6bb2
Expand message-related thread pools to 1 thread per shard.
2021-01-16 01:50:40 -05:00
4f9e7bb572
Separate Lettuce thread pools.
2021-01-16 01:18:05 -05:00
0a322d5a9f
Add a "doomsday switch."
2021-01-15 18:05:18 -05:00
59eb6d10c1
Gate based on destination rather than random.
2021-01-15 18:05:18 -05:00
a57ce1dd17
Add machinery to allow a percentage of message sends to succeed.
2021-01-15 17:05:16 -05:00
b100b3c36b
Reject traffic without logging exceptions
2021-01-15 16:23:53 -05:00
81c1ba6eef
Respond to all "message send" attempts with HTTP/503.
2021-01-15 15:34:14 -05:00
93ae4d1ee6
Move the client presence manager to its own breaker.
2021-01-15 13:51:39 -05:00
9c53d818f4
Use separate clusters for message cache read/write operations.
2021-01-15 13:51:39 -05:00
e5a2c1ab10
Always return an empty list of prekeys.
2021-01-15 12:27:10 -05:00
67ed035b36
Retry serializable key transactions.
2021-01-13 17:38:29 -05:00
ad30786f4a
Parallelize message persisters.
2021-01-12 18:50:14 -05:00
2e01da5ec1
Add a task to enable/disable accelerated crawling.
2021-01-11 19:29:18 -05:00
8fb37a0024
Log when a crawling cycle has wrapped up.
2021-01-11 19:29:18 -05:00
9412a7424c
Return HTTP/429 whenever somebody tries to get contacts from the old directory system.
2021-01-11 19:29:10 -05:00
f8cbb4f386
Temporarily suspend client version metrics to reduce load on our metric aggregator.
2021-01-11 14:04:44 -05:00
86ccaa52a5
Allow configuration of multiple directory account crawler listeners ( #325 )
...
* Allow configuration of multiple directory account crawler listeners
Only one should update the local redis directory. This one is marked
with replicationPrimary true. The others in the list only serve to
issue replication requests over to CDS replication load balancers.
* Update one more metric name
2021-01-10 17:11:02 -06:00
cc3e5d23e4
Enable Lettuce adaptive topology refreshes.
2021-01-10 16:20:35 -05:00
cac86d1f77
Standardize toplogy event handling strategy.
2021-01-10 15:14:12 -05:00
22f7bb822f
Raise log level of toplogy changes.
2021-01-10 15:14:12 -05:00
1b53f10091
Reload scripts across the whole cluster if one shard is missing the script.
2021-01-10 15:00:12 -05:00
bac268a21c
Don't send a reply to clients until messages are safely in a non-volatile store.
2021-01-10 13:03:40 -05:00
321e6e6679
Don't validate cluster membership (allow new shards to join dynamically).
2021-01-10 12:58:35 -05:00
22ef058cb6
Temporarily disable registration abuse system
2021-01-09 15:57:55 -05:00
9ee6419bc0
Publish directory updates to multiple SQS queues.
2021-01-08 18:07:18 -05:00
3bf0188e7f
Turn off alphanumeric sender ID for all countries.
2021-01-08 06:18:53 -05:00
91fc0fd623
Revert "Delete data in the storage service when deleting accounts."
...
This reverts commit ff1a721d5b
2021-01-08 06:18:39 -05:00
d2fcf68381
Record the status message when clients reject websocket messages.
2020-12-23 12:29:15 -05:00
a4d0c17efd
Record OS versions for iOS requests.
2020-12-23 11:36:31 -05:00
ff1a721d5b
Delete data in the storage service when deleting accounts.
2020-12-23 11:35:38 -05:00
c870a1bbd5
Introduce a storage service client.
2020-12-23 11:35:38 -05:00
ebf332a8c9
Record delivery duration excluding noise from non-primary devices ( #311 )
...
* Record delivery duration excluding noise from non-primary devices
* Extract method
2020-12-21 10:28:39 -06:00
85d1fff18f
Actually increment the Android request counter.
2020-12-11 11:46:07 -05:00
6bb106c2cb
Drop the Redis command timeout back down to 3 seconds to facilitate debug data collection.
2020-12-11 11:20:10 -05:00
e551fd2c1b
Revert "Pause checks for GV1 migration when checking for capability downgrades."
...
This reverts commit e7745db36e
2020-12-10 17:02:41 -05:00
34a11c2338
Record OS versions for desktop and SDK versions for Android.
2020-12-10 17:02:05 -05:00
0de3a400eb
Record unsuccessful server-to-client requests in more detail.
2020-12-10 17:01:46 -05:00
e524ff965d
Add a utility method for getting client platform tags from UA strings for metrics.
2020-12-10 17:01:46 -05:00
7ba689aaeb
Measure adoption of the `gv1-migration` capability.
2020-12-09 19:08:52 -05:00
92fde83b3a
Discard oversized messages bound for desktop clients via websockets.
2020-12-07 15:03:35 -05:00
3a268aef50
Reduce logging level for Lettuce connection events.
2020-12-07 11:56:41 -05:00
f673bd8d7b
Set device capabilities when linking a new device.
2020-12-02 13:21:08 -05:00
299b680013
Always include UUID in UD certificate ( #300 )
2020-12-01 08:56:55 -06:00
81e8352391
Time (and count) SQS "send message" operations.
2020-11-25 15:05:05 -05:00
1a627d6a87
Extend Redis command timeout to 3.5 seconds to avoid TCP retransmission "coincidences."
2020-11-25 15:04:06 -05:00
00a3e562dc
Force use of UCS-2 instead of GSM-7 for SMS to China ( #297 )
2020-11-20 14:41:48 -06:00
0628c9161c
Use named threads for the JsonMetricsReporter executor service.
2020-11-18 15:46:14 -05:00
9b28672e19
Honor disabled metric attributes in JsonMetricsReporter.
2020-11-18 15:46:14 -05:00
d764058a04
Measure contact intersection rate directly.
2020-11-18 14:28:53 -05:00
0aafe38496
Stop recording Lettuce latency metrics.
2020-11-17 13:20:37 -05:00
e7745db36e
Pause checks for GV1 migration when checking for capability downgrades.
2020-11-17 09:25:12 -05:00
474b879b16
Only notify CDS if an account attribute change actually changes an account's discoverability.
2020-11-16 10:54:12 -05:00
0a23b57ff8
Report Dropwizard metrics via the Wavefront proxy.
2020-11-13 17:14:13 -05:00
251e1b51c5
Make Micrometer batch size configurable.
2020-11-13 17:13:39 -05:00
2cc6c959a5
Revert "Temporarily suspend reporting of Lettuce latency metrics."
...
This reverts commit 2045153495a823b06334e7cbd86fb89c946c1cea.
2020-11-11 13:05:49 -05:00
fb9aa672c9
Include the name of the calling thread when a command times out.
2020-11-11 13:05:35 -05:00
325e65db7f
Expand UA parsing tests to cover OS details in desktop strings.
2020-11-11 13:05:18 -05:00
103b49ec45
Record the number of non-success responses from clients when sending messages via websockets.
2020-11-10 11:47:57 -05:00
6c78d7544f
Capture a thread dump when Redis commands time out.
2020-11-10 11:47:39 -05:00
7cf50a15d0
Include client age/UA string when closing due to a spurious keepalive request.
2020-11-10 11:45:12 -05:00
adbc4e9fec
Record the platforms of clients that send a keepalive without a local presence.
2020-11-10 11:45:12 -05:00
4815434dd7
Record the platforms of clients that are getting displaced.
2020-11-10 11:45:12 -05:00
b25e50bdae
Drop API keys from Micrometer configuration.
2020-11-09 09:26:56 -05:00
604287244f
Update copyright statement on all source files
...
IntelliJ Copyright Profile used to automate this.
2020-11-04 11:55:35 -05:00
4a4a721e90
Log timeouts in addition to incrementing a counter to make it easier to get precise timestamps.
2020-10-30 11:35:59 -04:00
a4062b338e
Count timeouts directly.
2020-10-29 10:51:18 -04:00
5587b7d469
Expose gv1-migration on profile endpoint
2020-10-28 13:00:57 -04:00
26870d134f
Set source UUID when delivering envelopes from message cache/db on websocket
2020-10-28 12:38:32 -04:00
0431a2abb1
De-dupe connection event logging messages.
2020-10-28 12:29:14 -04:00
c2db2d3cbd
Add GV1 Migration capability
2020-10-27 16:17:21 -04:00
05d9ec673e
Send push notifications if websockets close before all messages are delivered
2020-10-27 16:02:55 -04:00
1732cf9243
Add filters/tasks to enable/disable request logging.
2020-10-23 11:35:06 -04:00
ab62c19de9
Temporarily suspend reporting of Lettuce latency metrics.
2020-10-23 11:30:42 -04:00
96d3a69479
Use container-managed executors for APN/GCM senders.
2020-10-23 11:30:03 -04:00
8523bb1ad8
Change the "oversized message" threshold from 64kB to 1MB.
2020-10-23 11:13:19 -04:00
e6da54d9b8
Resolve build error introduced while merging.
2020-10-20 19:04:44 -04:00
0a843dc086
Tighten the "prune peers" interval; move from fixed-rate to fixed-delay scheduling.
2020-10-20 19:00:55 -04:00
7b3ed2dcbf
Catch exceptions thrown while pruning missing peers.
2020-10-20 19:00:55 -04:00
42ed6c3ded
Add clients to the "cleanup" list before actually setting their presence keys.
2020-10-20 19:00:55 -04:00
23ca011ac1
Record account deletion reasons.
2020-10-20 19:00:34 -04:00
d82b3dc429
Record a count of deleted accounts by country.
2020-10-20 19:00:34 -04:00
e391793c58
Remove now-redundant Redis execution time metrics.
2020-10-20 19:00:11 -04:00
236cef4b56
Report Lettuce command latency via Micrometer.
2020-10-20 19:00:11 -04:00
45687513bf
Revert "Revert "Share resources between Lettuce clients.""
...
This reverts commit 334f509be599fa6a501026e900d912ff7187e150.
2020-10-20 19:00:11 -04:00
019ffdaf12
Add a command for dumping Redis command stats.
2020-10-20 18:59:44 -04:00
df847431eb
Measure total bytes written to websockets and failed send attempts.
2020-10-20 17:22:30 -04:00
99f488d48f
Drop websocket connection names (unused for a while now).
2020-10-19 11:24:35 -04:00
05929871c9
Rename PushSender to MessageSender and add docs.
2020-10-19 11:24:35 -04:00
74b3daa70a
Collapse WebsocketSender into PushSender.
2020-10-19 11:24:35 -04:00
5e30b0499a
Move provisioning message-sending to its own manager class.
2020-10-19 11:24:35 -04:00
85c7347899
Add a command for dumping Redis SLOWLOG output.
2020-10-15 12:18:37 -04:00
3a84775912
Log cluster topology change events, too.
2020-10-13 16:07:08 -04:00
290a82e61c
Log when Lettuce connection events happen.
2020-10-13 16:07:08 -04:00
adac7d7fb2
Estimate the size of message entity lists sent via the REST API.
2020-10-13 15:49:11 -04:00
52320ebb91
Revert "Share resources between Lettuce clients."
...
This reverts commit eab1f503a5
2020-10-13 12:44:54 -04:00
eab1f503a5
Share resources between Lettuce clients.
2020-10-11 14:36:28 -04:00
a9d0aa136d
Add OS-reported metrics for cached/buffered memory.
2020-10-11 13:43:15 -04:00
691ab3080d
Fix some metrics names/types.
2020-10-11 12:37:17 -04:00
c5147e0c68
Report direct memory metrics.
2020-10-11 11:37:51 -04:00
e9b0829860
Report the maximum number of file descriptors allowed by the OS.
2020-10-11 11:27:57 -04:00
95428ab8b0
Report GC metrics.
2020-10-11 11:08:24 -04:00
775d56fe52
Drop the "repair message queue metadata" script.
2020-10-09 18:18:30 -04:00
ac2ff29288
Make sure to close scheduled reporters.
2020-10-09 18:05:00 -04:00
8e1975efe4
Record the number of deletable accounts per crawled chunk.
2020-10-08 10:51:41 -04:00
39c09733d3
Add /v1/payments/auth endpoint
2020-10-08 10:51:01 -04:00
e1c397993d
Require Android clients to support the gv2-3 capability
2020-10-06 16:49:49 -04:00
58ca4baf71
Time account deletion operations.
2020-10-06 11:04:47 -04:00
5245b68689
Remove temporary metrics.
2020-10-06 11:04:47 -04:00
2b6811cb1b
Really delete old accounts instead of just removing their push channels.
2020-10-06 11:04:47 -04:00
c82496b972
Remove the "repair queue metadata" script.
2020-10-05 16:57:16 -04:00
c31348ea9a
Drop the "insert messages" timeout.
2020-10-05 16:57:01 -04:00
c885540749
Check that the return of ZRANGEBYSCORE isn't an empty list.
2020-10-05 10:38:40 -04:00
bb087caddc
Don't panic if a queue exists, but is empty when repairing metadata.
2020-10-04 16:09:56 -04:00
5e3f8b9c2e
Disallow insertion of duplicate messages.
2020-10-04 15:34:14 -04:00
1ccfe928f7
Add a test to make sure that we don't double-insert messages with the same GUID.
2020-10-04 15:34:14 -04:00
3016269268
Revert "Temporarily disable the message persisters entirely."
...
This reverts commit d464721397
2020-10-04 15:25:06 -04:00
952cfae4e6
Repair queue metadata before persisting queues.
2020-10-04 15:25:06 -04:00
df7f209ebc
Revert "Don't insert message batches in transactions."
...
This reverts commit 16eefe333f
2020-10-04 15:12:15 -04:00
d464721397
Temporarily disable the message persisters entirely.
2020-10-04 11:44:35 -04:00
551a85c1e6
Use named variables instead of referring to KEYS/ARGV array indices in message cache scripts.
2020-10-04 11:27:27 -04:00
2686761608
Instrument "get queues to persist" calls and "persist queues" exceptions.
2020-10-04 10:48:42 -04:00
02a2c3224f
Discard unused feature flag constants/mocking.
2020-10-04 10:48:42 -04:00
8ec1dda9ba
Give the persister worker thread a meaningful name.
2020-10-04 10:48:42 -04:00
0308532523
Set a query timeout of 5 seconds when inserting batches of messages.
2020-10-04 10:48:42 -04:00
10b3af2947
Revert "Insert messages individually."
...
This reverts commit 158bfe4816
2020-10-04 10:48:42 -04:00
158bfe4816
Insert messages individually.
2020-10-03 13:13:34 -04:00
16eefe333f
Don't insert message batches in transactions.
2020-10-03 11:43:42 -04:00
65e585e122
Pause only if we're running low on queues to persist.
2020-10-03 11:43:34 -04:00
2ba36ee04c
Add a gauge for worker thread liveness.
2020-10-03 11:43:34 -04:00
fc05529574
Let MessagePersister manage its own worker thread.
2020-10-03 11:43:34 -04:00
07d24f487a
Don't re-register metrics for shared circuit breakers.
2020-10-02 15:05:00 -04:00
811acdb7f5
Use separate namespaces for Redis breaker/retry metrics.
2020-10-02 10:57:05 -04:00
a7266364d1
Refactor peer pruning to be more retry-friendly.
2020-10-01 17:17:07 -04:00
e83b41dc01
Reduce default Redis cluster command timeout to 3 seconds.
2020-10-01 17:17:07 -04:00
76665dd56e
Retry Redis commands that time out.
2020-10-01 17:17:07 -04:00
2d42b478ba
Consolidate cluster and pub/sub circuit breakers.
2020-10-01 17:17:07 -04:00
885fa6beae
Add tests for Device#isEnabled.
2020-10-01 12:54:35 -04:00
65cdd5fcbe
Drop the 365-day check when deciding if an account is enabled.
2020-10-01 12:54:35 -04:00
4302e19aba
Register a UUID argument factory for the messages database.
2020-10-01 11:06:43 -04:00
0c6f05f34a
Add a (failing!) test for sending a sealed-sender message after a non-sealed-sender message.
2020-10-01 11:06:43 -04:00
8040c285cd
Include stack traces when reporting persistence issues.
2020-09-30 11:47:16 -04:00
ada454f56f
Add a meter for persisting individual messages.
2020-09-30 10:39:56 -04:00
57d2ef8740
Return queues to the "to persist" list if something goes wrong during persistence.
2020-09-30 10:39:56 -04:00
a97e0982e3
Add an integration test for message persistence.
2020-09-30 10:39:56 -04:00
eaa2060d84
Fix an incorrect locking key and some previously-suppressed lock contention issues.
2020-09-30 10:39:56 -04:00
3e02c574e7
Log exceptions when persisting messages.
2020-09-30 10:39:56 -04:00
c7230ccbb0
Remove messages from the cache in bulk.
2020-09-29 10:58:02 -04:00
fc71ced660
Persist messages in batches.
2020-09-29 10:58:02 -04:00
6041a9d094
Make exit conditions slightly more conservative.
2020-09-29 10:58:02 -04:00
599cd766e1
Let Dropwizard manage persister thread lifecycles.
2020-09-29 10:58:02 -04:00
e64c8007c0
Detect GV2 capability in non-gcm Android devices
2020-09-28 15:54:10 -04:00
9339823e84
Add temporary metrics to monitor the ratio of enabled/disabled accounts.
2020-09-28 15:33:52 -04:00
e6d4620af1
Only allow linking desktop clients if they support the third-generation GV2 capability.
2020-09-25 17:08:32 -04:00
656e6db846
Only consider desktop devices GV2-capable if they send the third-gen GV2 capability.
2020-09-25 17:08:32 -04:00
30474e3a2b
Add a test for message ordering.
2020-09-25 11:41:58 -04:00
460bd98f1b
Add metrics for messages missing GUIDs.
2020-09-25 11:41:22 -04:00
a553eba574
Add an API endpoint for deleting accounts.
2020-09-25 11:39:17 -04:00
61f515670c
Add plumbing for deleting accounts and all associated data.
2020-09-25 11:39:17 -04:00
789af0f8a6
Add support for deleting keys associated with an account.
2020-09-25 11:39:17 -04:00
86fae58c96
Add support for deleting account entities from the database.
2020-09-25 11:39:17 -04:00
c54d3abe47
Check for the second-gen GV2 capability when linking devices.
2020-09-24 19:04:02 -04:00
6fe511eb50
Fix a bad size check when loading stored messages.
2020-09-23 18:02:33 -04:00
17d18b22c7
Drop pub/sub sending logic from WebsocketSender.
2020-09-23 14:51:02 -04:00
66a04ed730
Don't explicitly notify clients when messages get persisted.
2020-09-23 14:51:02 -04:00
7e14a0bc30
Drop pub/sub operations from WebsocketConnection.
2020-09-23 14:51:02 -04:00
77de0f86dc
Require desktop clients to send the new gv2-2 capability flag.
2020-09-23 12:05:58 -04:00
3b4bc9163a
Untangle thread pool names, tweak sizes, and add instrumentation.
2020-09-22 10:21:33 -04:00
e146135bd1
Don't attempt to send more messages if sending failed for any reason.
2020-09-22 10:21:33 -04:00
e9e18afb4a
Add a (failing) integration test demonstrating an infinite loop.
2020-09-22 10:21:33 -04:00
62c31eb202
Revert "Revert keyspace delivery for all messages"
...
This reverts commit 4dc49604b6
2020-09-22 10:21:33 -04:00
1eacee85ae
Count how many iOS users set the old GV2 capability flag.
2020-09-21 18:58:07 -04:00
5986145282
Add a second-generation GV2 capability and ignore the old capability for iOS devices.
2020-09-21 18:57:53 -04:00
b134a69a28
Record the number of authentications for users with/without GV2 support.
2020-09-21 15:42:13 -04:00
83f9eacac4
Refactor UserAgentTagUtil to parse UA strings with UserAgentUtil.
2020-09-21 12:24:08 -04:00
baab6b951b
Add a general utility class for parsing user-agent strings.
2020-09-21 12:24:08 -04:00
903a1bec91
Reject (eventually) oversize messages.
2020-09-17 17:07:20 -04:00
ebc3a251b7
Drop the UUID addressing capability flag entirely.
2020-09-14 15:36:29 -04:00
a567f4a6de
Don't check UUID capability when blocking capability downgrades.
2020-09-14 15:36:29 -04:00
4dc49604b6
Revert keyspace delivery for all messages
...
* Revert "Send all messages via keyspace notifications when a feature flag is enabled."
This reverts commit fadcf62166c02b25576689788fa665a052e2ee8f158e5004b76f9ff3be37f766c577438f53152c3e7bbc88d71668256d2343
2020-09-14 15:35:10 -04:00
fadcf62166
Send all messages via keyspace notifications when a feature flag is enabled.
2020-09-11 13:12:17 -04:00
c02b255766
Consolidate semaphore release logic.
2020-09-11 13:12:17 -04:00
89788fa665
Represent stored message state as an enumeration rather than a collection of booleans.
2020-09-11 13:12:17 -04:00
a052e2ee8f
Refactor: collapse state into semaphores/atomic booleans.
2020-09-11 13:12:17 -04:00
158e5004b7
Refactor: move sendNextMessagePage into its own method.
2020-09-11 13:12:17 -04:00
6f9ff3be37
Avoid querying the database if we think all new messages are in the cache.
2020-09-11 13:12:17 -04:00
f766c57743
Query for more stored messages if an update happens while we're already processing a batch.
2020-09-11 13:12:17 -04:00
8f53152c3e
Only send the "queue cleared" message once per websocket session.
2020-09-11 13:12:17 -04:00
7bbc88d716
Let processStoredMessages handle requery logic.
2020-09-11 13:12:17 -04:00
68256d2343
Only allow one thread to process stored messages at a time.
2020-09-11 13:12:17 -04:00
f88c440c48
Automatically retry when Twilio returns unreachable ( #190 )
...
* Parse and log the Twilio error code
* Automatically retry without sender ID when Twilio returns unreachable
* Remove attempt count and pass around whether or not sender id was used
2020-09-10 13:58:39 -05:00
cfa56ba6d4
Remove the "send online messages via keyspace notifications" feature flag.
2020-09-10 10:41:20 -04:00
2c6b646d87
Enforce no capability downgrade on device verification
2020-09-09 16:05:00 -04:00
e7572094b5
Require all enabled devices to support GV2.
2020-09-09 16:05:00 -04:00
5e34823a49
Optionally send online-only messages via keyspace notifications.
2020-09-09 14:42:09 -04:00
fdef21a871
Record and listen for ephemeral messages in a separate queue.
2020-09-09 14:42:09 -04:00
d40cff8a99
Revert "Add a system for storing, retrieving, and notifying listeners about ephemeral (online) messages."
...
This reverts commit 06754d6158
2020-09-08 15:55:09 -04:00
8927e45ded
Revert "Optionally send online-only messages via keyspace notifications."
...
This reverts commit 12fe28d8ab
2020-09-08 15:55:09 -04:00
1a93df92d4
Replace DeliveryStatus with a simple boolean.
2020-09-08 11:29:33 -04:00
12fe28d8ab
Optionally send online-only messages via keyspace notifications.
2020-09-08 11:19:55 -04:00
06754d6158
Add a system for storing, retrieving, and notifying listeners about ephemeral (online) messages.
2020-09-08 11:14:42 -04:00
1d5087374e
Jettison UUID-or-E164 plumbing in favor of UUID-only.
2020-09-08 09:30:47 -04:00
8356264fe0
Rename RedisClusterMessagesCache and related classes to just MessagesCache.
2020-09-08 09:30:47 -04:00
18ecd748dd
Entirely discard the old message cache machinery.
2020-09-08 09:30:47 -04:00
e324f27655
Stop sending/processing CONNECTED pub/sub messages.
2020-09-03 13:52:43 -04:00
afd645fb11
Retrieve messages using commands available in Redis 3.
2020-09-03 13:31:55 -04:00
5b42593fbb
Persist messages one page at a time.
2020-09-03 12:08:46 -04:00
5c04f2634a
Use a dedicated executor service for dispatching keyspace notifications.
2020-09-03 11:04:48 -04:00
ad01610d1e
Rely on the client presence manager to decide whether to send push notifications.
2020-09-03 11:04:48 -04:00
697c380cd1
Close websocket connections when displaced.
2020-09-03 11:04:48 -04:00
81e8143a43
Rely solely on the clustered message cache.
2020-09-02 11:57:33 -04:00
8409986ef5
Mirror persistence operations from the new persister to the old persister.
2020-09-02 11:02:40 -04:00
2b50367d7f
Put message persisters behind feature flags.
2020-09-02 11:02:40 -04:00
1dcc491fec
Move cache-mirroring operations to the calling thread.
2020-09-01 12:34:37 -04:00
d715f86713
Refactor to constants
2020-09-01 10:55:26 -04:00
5221828705
Increase maximum sticker size to 300 kibibytes
...
In preparation for animated stickers, allow stickers to be up to 300
kibibytes.
2020-09-01 10:55:26 -04:00
6aa4acd3db
Mirror "clear queue" operations to the clustered cache.
2020-09-01 10:55:07 -04:00
15936c29c1
Let Dropwizard manage the lifecycle of the feature flag manager.
2020-09-01 10:50:59 -04:00
8b70c69a0d
Replace metrics with logging statements.
2020-08-31 15:57:17 -04:00
dfe80a30dc
Make ScourMessageCacheCommand a ConfiguredCommand instead of an EnvironmentCommand.
2020-08-31 15:57:17 -04:00
ce026e7ad0
Don't send contacts to CDS if they've opted out of discoverability. (SERVER-130)
2020-08-27 15:58:02 -04:00
58e3122dab
Add a discoverableByPhoneNumber account attribute. (SERVER-129)
2020-08-27 15:58:02 -04:00
3b55b2d1b2
Actually make the "scour message cache" available to Dropwizard. Oops.
2020-08-27 15:15:04 -04:00
2326e61de5
Clear and re-create gauges to avoid "stuck" feature flag reporting.
2020-08-27 13:18:12 -04:00
32b18c9509
Add an endpoint for getting the current state of feature flags.
2020-08-27 13:18:12 -04:00
acf52ad8a3
Make feature flag manager tests use a real database to avoid over-mocking.
2020-08-27 13:18:12 -04:00
08dd493f98
Don't report exceptions as part of traffic metrics.
2020-08-27 13:17:57 -04:00
07bbe7dfb2
Return to an async model for push notification latency.
2020-08-27 10:51:44 -04:00
0aa1b80e3e
Add a command for persisting any detached messages in the old message cache.
2020-08-27 10:51:12 -04:00
5ac390281e
Add an abstract base class for Redis singleton tests.
2020-08-27 10:51:12 -04:00
ac465c5a18
Add a Lettuce-based Redis singleton client.
2020-08-27 10:51:12 -04:00
1ef3546822
Add support for server-side feature flags
2020-08-26 20:27:33 -04:00
e74ad2b555
Make RedisClusterMessagesCache a Managed class.
2020-08-25 10:58:01 -04:00
71c0056c66
Use lots of specific subscriptions instead of one monster subscription to minimize load.
2020-08-25 10:58:01 -04:00
56b27ea785
Record experiment outcomes with timers instead of counters.
2020-08-25 10:57:44 -04:00
2d75f59d33
Add support for UUID-only delivery certificates. (SERVER-132)
2020-08-20 17:05:53 -04:00
a709a3bcc0
Remove a candidate metric provider.
2020-08-20 15:40:56 -04:00
34bf5112e0
Drop TimeProvider.
2020-08-20 15:40:24 -04:00
bfe18d1d28
Re-nerf the clustered message persister.
2020-08-20 15:38:09 -04:00
6a76afc20d
Add a test to make sure the persister is respecting persist delays.
2020-08-20 15:38:09 -04:00
9c469c2f96
Base persister tests on a real Redis cluster.
2020-08-20 15:38:09 -04:00
2ab42f3dd6
Refine and expand clustered message cache metrics.
2020-08-19 11:39:05 -04:00
af34b43a8d
Reactivate the message notification experiment.
2020-08-19 11:39:05 -04:00
0f71cc7864
Rename metrics associated with cluster circuit breakers for clarity.
2020-08-18 17:59:00 -04:00
df90de3a5f
Change default Lettuce command timeout to 10s.
2020-08-18 16:21:42 -04:00
42ea7a9814
Revert Lettuce connection pooling.
2020-08-18 16:21:42 -04:00
c683cbdb2d
Time Redis operations.
2020-08-18 12:20:12 -04:00
d243b73678
Make Lettuce connection pools configurable. Double the default size.
2020-08-18 12:20:12 -04:00
dc28d063aa
Reactivate the explicit client presence experiment.
2020-08-17 11:34:27 -04:00
bb6045c1d0
Disarm the client presence manager experiment.
2020-08-15 20:23:05 -04:00
f1a74b5939
Disarm new message keyspace notifications.
2020-08-15 20:23:05 -04:00
6fb9038af1
Move to a synchronous, pooled connection model for Redis clusters.
2020-08-14 17:15:56 -04:00
27f721a1f5
Update to resilience4j 1.5.0.
2020-08-14 17:15:56 -04:00
5717dc294e
Combine the read/write breakers for Redis clusters.
2020-08-14 17:15:56 -04:00
ae0f8df11b
Break out FaultTolerantPubSubConnection as its own thing so different use cases can have their own subscription space.
2020-08-14 17:15:56 -04:00
77460ba502
Remove keyspace notification configuration checks because AWS doesn't support `CONFIG GET`.
2020-08-13 15:32:25 -04:00
f8235da4d8
Fix an issue where the queue for a thread pool was not bounded.
2020-08-13 12:46:11 -04:00
8d3316ccd6
Listen for new messages via keyspace notifications.
2020-08-13 12:17:04 -04:00
2c29f831e8
Add an explicit client presence system.
2020-08-13 10:56:26 -04:00
9457325119
Add pub/sub affordances to FaultTolerantRedisCluster.
2020-08-13 10:56:26 -04:00
189f8afcc9
Warm up the test cluster before running tests to avoid transient startup jitters.
2020-08-13 10:56:26 -04:00
9699b67510
Record the size of outgoing message lists.
2020-08-12 16:57:10 -04:00
d60633a46c
Add a meter for the number of messages we send via websocket connections.
2020-08-12 16:57:10 -04:00
0fcf28e7e7
Use the MessagesManager to actually persist messages.
2020-08-11 15:50:22 -04:00
5fad8f74b1
Factor MessagePersister into its own class.
2020-08-11 15:50:22 -04:00
e35e34d2e0
Move operation-mirroring logic to MessagesManager.
2020-08-11 15:50:22 -04:00
31a215d4d6
Use "global." instead of "g." as the prefix for global config options.
2020-08-11 11:55:35 -04:00
30948de13d
Update a metric provider dependency and remove a workaround for an upstream issue.
2020-08-11 11:02:38 -04:00
b97158bf7b
Create global remote config controllable in the signal server configuration ( #127 )
...
* Add global config controller through file rather than database
* Do no permit attempting to set or delete global config entries
2020-08-10 16:31:15 -05:00
6646be8d94
Make CpuUsageGauge a CachedGauge.
2020-08-10 12:56:37 -04:00
647a2aea64
Cache a reference to the OS management bean to avoid repeated lookups.
2020-08-10 12:56:37 -04:00
58e58ce51c
Remove a candidate metric provider.
2020-08-10 11:03:20 -04:00
4b7e48d3ec
Override default ingestion URI for SignalFx ( #131 )
2020-08-07 15:29:42 -05:00
0e074d3a5a
Copy SignalFxMeterRegistry into a new class to get better logging
2020-08-07 16:01:56 -04:00
ea00224e7f
Add support for reporting metrics to signalfx ( #129 )
2020-08-07 11:10:31 -05:00
38293efe75
Keep a running count of the number of open websockets.
2020-08-06 16:07:34 -04:00
3286c5e174
Disable Redis persistence for tests.
2020-08-06 11:22:51 -04:00
e0f8a28f38
Close connections before closing the whole cluster client.
2020-08-06 11:22:31 -04:00
bf1b00b163
Drop a spurious RedisClusterClient.
2020-08-06 11:22:31 -04:00
4fa3a136ad
Remove arbitrary SMS and add a NANPA message service ( #123 )
...
* Remove arbitrary SMS code
This code has run its course and is no longer needed for now.
* Add elements to sample config that were left out
* Add a messaging service for NANPA
* Fixup sample config capitalization
2020-08-05 13:35:11 -05:00
178a6bd66e
Log the top-level exception name and message when crawling badness happens.
2020-08-05 11:23:16 -04:00
57e1339230
Further restrict user agent pattern matching ( #120 )
...
* Further restrict user agent pattern matching
* Add static qualifier to method
2020-08-04 12:58:16 -05:00
4144423227
Publish percentiles for Micrometer distributions/timers.
2020-08-04 10:58:59 -04:00
4d03514142
Add a command for clearing the messages cache cluster.
2020-08-04 10:58:41 -04:00
0bc5566976
Mirror delete-after-persist operations to the clustered message cache.
2020-08-04 10:58:41 -04:00
925567add5
Actually "plug in" the reglock counter.
2020-08-03 15:43:33 -04:00
ad97731d46
Reduce the maximum number of versions in play to 1,000.
2020-08-03 15:42:15 -04:00
40684a93a2
Restrict user-agent version matching to a more confined space.
2020-08-03 15:42:15 -04:00
f3b644ceb8
Update the push latency manager to use UUIDs and a Redis cluster.
2020-08-03 15:36:02 -04:00
901ba6e87f
Added a push latency manager.
2020-08-03 15:36:02 -04:00
76389bd584
Clear would-be-persisted messages from the cache cluster, but don't store them to the database.
2020-07-30 19:14:39 -04:00
7bf8650d59
Un-manage FaultTolerantRedisCluster so it shuts down at JVM shutdown instead of Jetty shutdown.
2020-07-30 18:37:38 -04:00
7cb24dd96d
Add environment tag to datadog metric reporting
2020-07-30 18:04:16 -04:00
dee040318a
Add the host tag to datadog metric reporting
2020-07-30 18:04:16 -04:00
baf563e46d
Temporarily disarm the actual persisting part of the message persister.
2020-07-30 17:12:37 -04:00
e10246f10b
Use Dropwizard timers/histograms for persister metrics.
2020-07-30 14:27:06 -04:00
a9dfd88671
Start the clustered message persister at application startup.
2020-07-30 12:32:35 -04:00
beac73b6c8
Add a cluster-capable message persister
2020-07-30 11:39:14 -04:00
f9f93c77e2
Use UUIDs instead of phone numbers as account identifiers in clustered message cache
2020-07-30 11:39:14 -04:00
6fc1b4c6c0
Add a cluster-backed message cache.
2020-07-30 11:39:14 -04:00
639898ec07
Expand Experiment to deal with async suppliers and Optionals.
2020-07-30 11:39:14 -04:00
3d3790fdbc
Add binary execution methods to ClusterLuaScript.
2020-07-30 11:39:14 -04:00
69c8968cb0
Add byte-array-based methods to FaultTolerantRedisCluster.
2020-07-30 11:39:14 -04:00
aa25fc7901
Fix UsernamesManager metric/logger names.
2020-07-29 11:00:29 -04:00
4aba493ee2
Fix the key used for database crawler workers.
2020-07-29 10:58:06 -04:00
b9cfac5934
Introduce additional metric aggregators.
2020-07-28 15:11:51 -04:00
f8e97fcc32
revise 12 hour active user fudge to 8 hours for better continuity of data from a month ago
2020-07-28 11:09:41 -07:00
7f8f2641f6
Simplify registration lock counting by avoiding inactive accounts.
2020-07-28 11:48:20 -04:00
022dbb606f
Count registration lock versions when crawling the account database.
2020-07-28 11:48:20 -04:00
fea72b190d
Record message content size as a dimensioned distribution.
2020-07-28 11:47:56 -04:00
eea073f882
Decommission the old cache.
2020-07-28 10:29:28 -04:00
fc1d88f5bb
Read exclusively from the cache cluster.
2020-07-27 15:11:40 -04:00
acbe410e0b
Remove a metric aggregator.
2020-07-27 12:50:49 -04:00
89bafea61f
Move SMS strings to configuration
2020-07-27 11:23:21 -05:00
33a0c4a9ae
Use first party metric aggregator libraries where possible.
2020-07-24 17:21:56 -04:00
4cc5999f05
Configure additional metric aggregators.
2020-07-23 13:31:19 -04:00
0fbf31ec98
Clear each cluster node individually.
2020-07-22 11:12:21 -04:00
db9b7ca447
Fix slot assignment when building a cluster for tests.
2020-07-22 11:04:10 -04:00
eecc71c77f
Revert batch message storage. ( #95 )
2020-07-20 16:28:32 -04:00
5f898a9071
Measure inserted message batch size.
2020-07-20 10:30:29 -04:00
a08f21336a
Be explicit about transaction management.
2020-07-20 10:30:29 -04:00
215125de26
Update tests.
2020-07-20 10:30:29 -04:00
dfa94eac41
Store messages in batches.
2020-07-20 10:30:29 -04:00
247d869a5c
De-randomize message tests to minimize flakiness.
2020-07-14 18:46:39 -04:00
b9b6e1818f
Rename SenderIdSelector to SenderIdSupplier per code review discussion
2020-07-14 10:53:48 -05:00
a7968ccc3c
Address code review comments
2020-07-14 10:53:48 -05:00
b7e0e5a356
Create a strategy class to decide which sender id to use
...
The rules around selecting sender ids can get complicated with some
countries not supporting it and others requiring pre-registration that
may result in having a different sender id for that country than
others. This strategy class handles the logic of dealing with this
expanded configuration and applying the appropriate sender id or none
when it's not appropriate to do so at all.
2020-07-14 10:53:48 -05:00
e3aecb2aa9
apply a 12 hour fudge to daily user counting to account for last seen timestamp fuzzing
2020-07-09 17:43:12 -07:00
116ab83b95
Include a PushType header when sending APNs notifications.
2020-07-09 16:12:20 -04:00
c5d0d4acd0
Revert "Move rate limiter logic to Lua scripts"
...
This reverts commit b585c6676d
2020-07-09 12:30:25 -04:00
06190286ec
Remove temporary circuit breaker suppression.
2020-07-07 16:33:05 -04:00
3bca856e87
Remove a pair of spurious SET calls in the rate limiter script.
2020-07-07 16:33:05 -04:00
b3a778b89a
Temporarily catch and log all script execution exceptions to avoid opening the breaker.
2020-07-07 15:17:25 -04:00
dcb11f7606
Log errors from experiments.
2020-07-07 15:17:25 -04:00
933ce42d5a
Test rate limiters against a real cluster.
2020-07-07 15:17:25 -04:00
6c1ba957bd
Ensure the default alphaId configuration is an empty list rather than null
2020-07-07 10:17:40 -05:00
e021286eee
Add configuration by country for sending from alpha IDs
2020-07-07 10:17:40 -05:00
0ee7a66033
Keep trying ports until you get one lower than 55535 ( #83 )
...
* Keep trying ports until you get one lower than 55535
* Rename method and change to do...while
* Limit attempts to 11,000 to find an open redis cluster port
2020-07-07 10:12:31 -05:00
42c797ee97
Set the default log level for tests to WARN.
2020-07-07 11:05:39 -04:00
b585c6676d
Move rate limiter logic to Lua scripts
2020-07-06 10:10:13 -04:00
f5ddb0f1f8
Test ClusterLuaScript against a real Redis cluster.
2020-07-02 18:58:30 -04:00
ef97f9e738
Revert "Temporarily suspend execution of the "unlock" script."
...
This reverts commit 6aecd8d44a
2020-07-02 18:58:30 -04:00
26a03b55de
Un-reinvent the clustered script execution wheel.
2020-07-02 18:58:30 -04:00
b93a16abae
Honor the step size set in the micrometer config.
2020-07-02 11:40:41 -04:00
ff2783d434
Fixed a goof where we were mirroring a write to the wrong key in the new cache cluster.
2020-07-02 11:40:27 -04:00
25a5a8db68
Set avatar to null on Account when request is false ( #78 )
2020-06-29 15:53:31 -05:00
a68d91b54c
Resolve some test flakiness by adding a deterministic "wait" mechanism. (SERVER-86)
2020-06-29 12:24:25 -04:00
88ec3a5751
Add a counter for dead letter events.
2020-06-26 09:00:11 -04:00
734dc2e37a
Don't block the Redis instance when clearing the cache.
2020-06-19 10:52:18 -04:00
6aecd8d44a
Temporarily suspend execution of the "unlock" script.
2020-06-17 22:27:02 -04:00
bbf5e1fa78
Use the UA string from websocket upgrade requests if available.
2020-06-17 15:40:18 -04:00
7454e55693
Write synchronously to the cache cluster.
2020-06-17 15:38:56 -04:00
c745fe7778
Fix a poorly-mirrored cache delete operation.
2020-06-17 15:35:46 -04:00
6adcebb247
Return to just using counters instead of timers for measuring experiment outcomes.
2020-06-17 15:34:02 -04:00
38f9b8f3dd
Make write operations in `AccountDatabaseCrawlerCache` synchronous.
2020-06-17 10:05:43 -04:00
7faf143a97
Subdivide the account database crawler cache experiment and add logging to track down lingering disagreements.
2020-06-17 09:23:40 -04:00
17cfd4924c
Fixed a poorly-mirrored write operation to the new cluster.
2020-06-16 16:46:41 -04:00
a0bebca1e6
Extend Experiment to report more detail when results don't match.
2020-06-16 16:46:41 -04:00
75cbfa2898
Mirror unlock-via-script calls to the cache cluster.
2020-06-16 16:46:41 -04:00
58a8ed1588
Add a cluster-friendly version of LuaScript.
2020-06-16 16:46:41 -04:00
e032f8df59
Add a command for clearing the cache cluster.
2020-06-16 16:46:41 -04:00
b16e37d80a
Record a histogram of incoming message list sizes.
2020-06-12 14:43:50 -04:00
c17cc07b73
Instrument BlockingThreadPoolExecutor.
2020-06-12 14:43:50 -04:00
6f767a72a7
Add a timer for the private sendMessage method.
2020-06-12 14:43:50 -04:00
11196436e9
Time rate limiter validation calls.
2020-06-12 14:43:50 -04:00
9afc433db4
Record exceptions associated with server responses.
2020-06-11 22:08:07 -04:00
f701e3d834
Record distributions of timer values; stop recording error causes.
2020-06-11 11:50:36 -04:00
4c623ca3c5
Compare Redis reads using Lettuce's synchronous path.
2020-06-11 11:50:36 -04:00
0671f05c05
Introduce experiment comparison methods for suppliers.
2020-06-11 11:50:36 -04:00
0713da7393
Record experiment results with a timer instead of a counter.
2020-06-11 11:50:36 -04:00
05955d0483
Check for null header values before trying to iterate through them.
2020-06-09 15:45:32 -04:00
28c765bd9a
Add an in-app-context test for websocket metrics.
2020-06-09 15:45:32 -04:00
8287317be7
Add account device ID to the prekey rate limiter
...
This limits prekey fetching per device on an account instead of on an
account level.
2020-06-09 10:20:10 -07:00
ec858b2d4c
Set a timeout for Redis cluster operations and shut down the cluster as part of service shutdown
2020-06-07 18:27:57 -04:00
47ece983d2
Added a Redis cluster health check.
2020-06-07 18:27:11 -04:00
52310b5dd9
Compare results of reads from old and new Redis caches.
2020-06-07 18:27:11 -04:00
c2a4a2778e
Introduce the Experiment class to compare results from parallel systems.
2020-06-07 18:27:11 -04:00
1db5977e80
Mirror username deletes unconditionally.
2020-06-07 18:27:11 -04:00
1b5dc0e434
Fixed a potential issue where locks could get out of sync between Redis instances.
2020-06-07 18:27:11 -04:00
f07f02d866
Deliver upgrade link to stale clients
2020-06-06 18:20:55 -07:00
1388103919
Mirror writes to the cache cluster.
2020-06-06 20:37:48 -04:00
fe1054d58a
Introduce a Lettuce-based fault-tolerant Redis cluster accessor.
2020-06-06 20:37:48 -04:00
ba6ac778fc
Update to Pushy v0.14.1.
2020-06-05 12:21:56 -04:00
228ffcbfce
Differentiate between websocket and "boring" HTTP traffic.
2020-05-28 12:52:49 -04:00
f18ab9e5cc
Measure traffic from websockets.
2020-05-28 12:52:49 -04:00
06c82ee87d
Celebrate the diversity of UA strings when generating tags for metrics.
2020-05-27 19:35:42 -04:00
9ba5ee8043
Move UA tag extraction into its own utility class.
2020-05-27 19:35:42 -04:00
eede4e50ca
Use hashed UUID to spread last seen updates over a full day ( #40 )
2020-05-26 13:38:52 -07:00
aa10f63d9f
Add the timestamp using the `add` method.
2020-05-22 17:39:25 -04:00
a25af36e32
Include timestamps in all server-to-client websocket messages.
2020-05-22 15:13:39 -04:00
817f057927
Inject timestamps into responses.
2020-05-22 15:13:39 -04:00
a13c44d81a
Capture request-level metrics (path, status, client platform/version).
2020-05-20 17:48:19 -04:00
45ad8f8ffb
Add the Wavefront/Micrometer reporter as a dependency and configure a registry.
2020-05-20 17:46:07 -04:00
7da9e88c0b
Add hashKey to RemoteConfig
...
This allows the percentages for different entries in remote config to
be aligned so one remote config can be a subset of another.
2020-05-13 11:08:22 -07:00
1c73c91133
Report the number of days until the CDS CA cert expires as a metric so we can set an alarm.
2020-05-12 12:57:11 -04:00
b1d11d4f69
Use APNs signing keys instead of expiring certificates.
2020-05-12 12:48:28 -04:00
001a9310c3
Support device transfers (SERVER-41, SERVER-42) ( #32 )
...
This change introduces a `transfer` device capability and account creation argument in support of the iOS device transfer effort.
2020-05-12 12:23:18 -04:00
8ffadfa1f1
Add payment addresses on account attributes update
2020-05-07 09:52:38 -07:00
50d7929e76
Drop the GCM `RECEIPT` message type (unused).
2020-05-04 17:51:54 -04:00
10840b22c5
Don't let one unregistered device block receipt for others.
2020-05-04 17:51:25 -04:00
acfbab5915
Update to Pushy v0.13.11.
2020-05-04 17:50:35 -04:00
48c324fe86
Use a static sequence of randomness in tests
...
The RemoteConfigControllerTest#testMath unit test would occassionally
fail because randomness doesn't necessarily group into expected ranges
over a finite trial count. This changes the test to use a predefined
PRNG sequence instead of one that varies with each test so that the
test will no long randomly fail.
2020-04-29 17:31:43 -07:00
0c495e7e72
Workaround lack of internal retry on transaction rollback
...
The get endpoint for key fetching can fail if the transaction cannot
complete because of simultaneous modification. Clients currently
receive 500 from this and retry if it happens, but this test case runs
into it without retrying and then complains that not all the threads
completed successfully. This workaround adds some retry attempts.
2020-04-29 17:10:13 -07:00
50ccfee201
Allow remote config to send non-boolean values
...
This version of remote config allows non-boolean values to be returned
to clients but unfortunately limits the configuration to only one
value or another. There is no way to configure more than two values
for the same key with this setup.
2020-04-29 10:51:10 -07:00
95f0ce1816
Support for advertising payment addresses on profile
2020-04-22 12:32:53 -07:00
a32c8fabed
Temporarily move GV2 capability from allMatch to anyMatch
2020-04-20 13:42:36 -07:00
1408ac77f9
Make storageCapable a boolean result rather than an auth token
2020-04-09 10:19:49 -07:00
7e97d10ae1
Fix account dropping new style registration locks
2020-04-06 09:27:23 -07:00
56b134facd
Change attachment key from long to base64 of 15 bytes
2020-04-02 10:20:42 -07:00
41286650cc
Create attachments V3 endpoint for CDN2 on GCP
...
In preparation for resumable uploads, this creates a separate
attachment authorization endpoint that creates a signed URL for
accessing GCP Storage through Signal's CDN2. This should allow Signal
clients to do byte-level resume of media uploads.
2020-04-02 10:20:42 -07:00
3c8e7c6c10
Add storage capability and return KBS creds on rereg w/ storage set
2020-03-27 10:45:48 -07:00
4f64513c83
Break out redis pubsub into dedicated cluster
2020-03-16 17:44:42 -07:00
350f5ccb3c
Account for fronted regions
2020-03-14 19:07:42 -07:00
ac1153c7cf
Additional limits
2020-03-14 18:10:07 -07:00
3b1672a4a7
Update zkgroups to 0.4.0
2020-03-14 16:30:13 -07:00
009f81a9a6
Update to dropwizard 2.x
2020-03-14 16:30:13 -07:00
8b10b1dc62
Remove tombstone column from keys table
2020-02-25 12:25:34 -08:00
077c259d5b
Migrate keys to accountsdb
2020-02-23 17:59:30 -08:00
e5746c19cf
Support for GV2 capability flag
2020-02-07 11:53:28 -08:00
e399f9e851
Generate external creds for KBS based on UUID
2020-01-22 13:47:33 -08:00
e4e20c2d25
Add support for UUID buckets in remote config
2020-01-22 11:28:08 -08:00
08a70664f4
Support for getting/setting remote config variables
2020-01-21 13:38:58 -08:00
75fc35ee4b
Parameterize access to zk operations
2020-01-21 11:29:08 -08:00
ba3102d667
Support for versioned profiles
...
Includes support for issuing zkgroup auth credentials
2020-01-21 11:04:06 -08:00
8a9fed64f2
Support for first/last profile name length
2020-01-13 18:55:04 -08:00
71c7e30548
Increase max size for sticker manifest
2019-12-19 10:29:47 -08:00
886db1a2c3
Bump max sticker count to 201
2019-12-18 17:08:51 -08:00
b4c06db031
Make redis failures on write-back retrieve non-fatal
2019-11-20 12:36:22 -08:00
82486a873a
Delete old username mapping when setting new one
2019-11-20 12:36:22 -08:00
99760ba6a0
Put UUID on server-generated delivery receipt
2019-11-20 12:36:22 -08:00
2b987e6e93
Usernames can't start with numbers
2019-11-20 12:36:22 -08:00
523134f24b
Username reservation table
2019-11-20 12:36:22 -08:00
99c228dd6d
Support for setting and looking up usernames
2019-11-20 12:36:22 -08:00
44d38a00d4
Fix capabilities NPE
2019-11-14 13:36:40 -08:00
c623f70caa
Add support for capabilities
2019-11-14 13:36:40 -08:00
f16b783378
return backup, not storage, credentials for reg lock
2019-11-05 10:36:33 -08:00
be4b75932b
since onCrawlChunk() is now protected, we need to invoke timeAndProcessChunk() in our unit tests
2019-10-29 18:20:03 -07:00
04d7f3a5dc
allow disabled accounts to get KBS auth
2019-10-29 16:50:47 -07:00
eddfacd0f4
add timers to the account crawler listeners
2019-10-25 21:30:48 -07:00
69742839c0
uuid-based account crawler
2019-08-27 14:42:14 -07:00
20b5f0e681
Reset cache index
2019-08-27 14:08:50 -07:00
3803b8f284
Fix for jedis pool deadlock
...
1) Remove nested pool checkouts
2) Add a max wait so it won't block forever on deadlock
2019-08-27 14:02:42 -07:00
e3daf743f2
Fix new account calculation
2019-08-27 11:14:11 -07:00
ae5da74bb1
Update banner
2019-08-26 16:08:30 -07:00
cf78047830
revert to phone number-based account crawler
2019-08-26 14:00:15 -07:00
284428a45a
Support for authentication to KBS
2019-08-26 11:09:54 -07:00
79f2efdfd9
Make UUID in sealed sender certificate optional for buggy clients
2019-08-26 11:09:54 -07:00
07822b371f
replicate uuids to contact discovery
2019-08-26 11:09:54 -07:00
7a3a385569
Support for UUID based addressing
2019-08-26 11:09:54 -07:00
e57f78cf90
Add meter for GCM challenge transmissions
2019-08-01 13:30:49 -07:00
10724fee04
Support for sticker pack uploads
2019-07-24 16:29:56 -07:00
4d09bae09b
Add some logging
2019-07-11 19:57:31 -07:00
11902dec3c
Support for v2 registration lock
2019-07-11 18:15:14 -07:00
4fdbe9b9ff
Support for push preauth
2019-07-11 18:15:10 -07:00
a6e7e30177
Add requester to recaptcha validation
2019-07-11 17:31:34 -07:00
5b69ff7e94
Break out keys database and accounts database
2019-06-19 17:16:37 -07:00
bc0c6be4c5
We don't need to support disabled accounts for the signed PK API
2019-06-12 12:32:15 -07:00
3c6b418ca8
Publish fcm retry metrics
2019-05-30 11:05:05 -07:00
105a38a7db
Update gcm-sender-async to use jdk11 httpclient
2019-05-30 10:46:40 -07:00
e6f25b9c5e
Bring gcm-sender-async in as a module
2019-05-29 11:03:33 -07:00
6e0b956e61
Only set the uninstall feedback timestamp when it's zero
...
Otherwise each send will update the timestamp, preventing it from
aging out to the point where the cleaner will pick it up.
2019-05-26 14:27:30 -07:00
a029768d24
Reenable account cleaner
2019-05-10 10:42:42 -07:00
4d9c9206cf
Delay processing FCM uninstalled feedback
...
Check to make sure client is not still active before unregistering,
since FCM feedback seems to be often erroneous
2019-05-07 10:04:22 -07:00
35116f9229
Clean up concepts of enabled account state
...
1) Rename "active" methods to be "enabled," since they aren't
really about "activity."
2) Make authentication fail if a device or account is in dissabled
state.
3) Let some controllers authenticate accounts that are in a
disabled state.
2019-05-04 12:31:50 -07:00
a1f90cd39b
Temporarily disable account cleaner
2019-05-03 12:09:01 -07:00
45dc7459b8
Temporarily disable GCM unregistered feedback
2019-05-03 11:51:21 -07:00
6877b663f1
enable up to 40 account updates per chunk in AccountCleaner
2019-05-03 10:58:57 -07:00
3c69f81a10
expire accounts explicitly
2019-05-02 21:14:57 -07:00
d316d57e5d
fix DirectoryController tests
2019-05-02 19:20:23 -07:00
92eddf8eb6
Directory feedback v3
2019-05-02 15:49:27 -07:00
0c81556b90
Switch websocket-resources from ListenableFuture to CompletableFuture
2019-05-02 15:05:44 -07:00
d0d375aeb7
Break out into a multi-module project
2019-05-01 13:19:11 -07:00
Chris Eager