Commit graph

1498 commits

Author SHA1 Message Date
Jeong Arm
1de748bf5e
Fix FetchFeaturedCollectionService (#18030) 2022-04-13 13:25:42 +02:00
Claire
ff73d43b35 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `Gemfile.lock`:
  Not a real conflict, just an upstream dependency udpated
  textually too close to a glitch-soc-only dependency.
  Updated dependencies like upstream.
- `app/controllers/settings/preferences_controller.rb`:
  Upstream added settings where we had extra glitch-soc-specific settings.
  Added upstream's new settings.
- `app/models/user.rb`:
  Upstream added settings where we had extra glitch-soc-specific settings.
  Added upstream's new settings.
- `config/i18n-tasks.yml`:
  Not a real conflict, just a new upstream line too textually close to
  a glitch-soc-only line.
  Ported upstream's change.
2022-04-08 19:53:32 +02:00
Eugen Rochko
8e20e16cf0
Change e-mail notifications to only be sent when recipient is offline (#17984)
* Change e-mail notifications to only be sent when recipient is offline

Change the default for follow and mention notifications back on

* Add preference to always send e-mail notifications

* Change wording
2022-04-08 18:03:31 +02:00
Claire
b368c75029 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/controllers/concerns/sign_in_token_authentication_concern.rb`:
  Upstream removed this file, while glitch-soc had changes to deal with
  its theming system.
  Removed the file like upstream did.
2022-04-06 21:10:23 +02:00
Claire
8f91e304a5
Fix spurious edits and require incoming edits to be explicitly marked as such (#17918)
* Change post text edit to not be considered significant if it's identical after reformatting

* We don't need to clear previous change information anymore

* Require status edits to be explicit, except for poll tallies

* Fix tests

* Add some tests

* Add poll-related tests

* Add HTML-formatting related tests
2022-04-06 21:01:02 +02:00
Claire
454ef42aab
Fix error when encountering invalid pinned posts (#17964) 2022-04-06 20:58:23 +02:00
Claire
abb11778d7
Fix inconsistency in error handling when removing a status (#17974)
Not completely sure this could actually have any ill effect, but if
`RemoveStatusService` fails to acquire a lock in an
`ActivityPub::ProcessingWorker` job processing a `Delete`, the status
is currently discarded and causes a job failure but the next time the
job is attempted, it will skip deleting the status due to it being
discarded.

This commit makes the behavior of `RemoveStatusService` a bit more consistent
in case of failure to acquire the lock.
2022-04-06 20:57:52 +02:00
Claire
51a297e128 Merge branch 'main' into glitch-soc/merge-upstream 2022-03-30 13:40:10 +02:00
Claire
2cc7ba2671
Refactor response_to_recipient? CTE (#17899)
* Optimize and clean up `response_to_recipient?` CTE

Marginally improve performances, and make the CTE much more readable

* Limit max depth in `response_to_recipient?` CTE
2022-03-30 10:26:51 +02:00
Claire
aaa9ec340b Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/lib/formatter.rb`:
  Upstream completely refactored the formatting code and removed that file,
  while glitch-soc had code for Markdown and HTML toots.
  Took upstream code, glitch-soc changes will be re-implemented on top of the
  refactored classes in a later commit.
- `app/models/status.rb`:
  Upstream refactored status edit handling and moved code to
  `app/models/concerns/status_snapshot_concern.rb`.
  Applied glitch-soc's changes to that file.
- `app/serializers/activitypub/note_serializer.rb`:
  Not really a conflict, just a line added too close to one modified by
  glitch-soc.
  Applied upstream changes while keeping the glitch-soc-modified one.
- `app/services/update_status_service.rb`:
  Not really a conflict, upstream modified a line adjacent to one added by
  glitch-soc.
  Applied upstream changes while keeping the glitch-soc line.
- `app/views/statuses/_simple_status.html.haml`:
  Upstream refactored formatting, glitch-soc changed the markup slightly.
  Applied upstream changes.
- `spec/lib/formatter_spec.rb`:
  Upstream completely refactored the formatting code and removed that file,
  while glitch-soc had code for Markdown and HTML toots.
  Took upstream code, glitch-soc changes will be re-implemented on top of the
  refactored classes in a later commit.
2022-03-26 19:18:55 +01:00
Eugen Rochko
cefa526c6d
Refactor formatter (#17828)
* Refactor formatter

* Move custom emoji pre-rendering logic to view helpers

* Move more methods out of Formatter

* Fix code style issues

* Remove Formatter

* Add inline poll options to RSS feeds

* Remove unused helper method

* Fix code style issues

* Various fixes and improvements

* Fix test
2022-03-26 02:53:34 +01:00
Eugen Rochko
71f2b95106
Fix edits with no actual changes being allowed (#17843)
* Fix edits with no actual changes being allowed locally

* Fix edits with no actual changes being allowed through ActivityPub

* Fix false positive changes caused by description processing in model

* Fix not recording poll expiration update

* Fix test

* Revert changes to ProcessStatusUpdateService

* Various fixes and improvements

* Fix code style issues

* Various changes and improvements

* Add guard clause
2022-03-26 00:38:44 +01:00
Claire
8d4b5ee3ef Merge branch 'main' into glitch-soc/merge-upstream 2022-03-12 20:20:58 +01:00
Claire
92a86b958e
Fix issues with processing toot edits (#17748)
* Fix searching for an already-known status by URL not working

* Fix Update processing from statuses prior to 20220302232632

`ordered_media_attachment_ids_changed?` would return `true` when going from
`nil` to anything (including `[]`).

* Add tests
2022-03-12 19:33:10 +01:00
Claire
bb9a6fb5c2 Merge branch 'main' into glitch-soc/merge-upstream 2022-03-12 11:51:25 +01:00
Claire
d3aa9cf774
Fix Updates being forwarded even when not processable or causing no change (#17699)
* Fix Updates being forwarded even when not processable or causing no change

* Refactor and ensure status edit is strictly newer than last known edit
2022-03-12 09:30:31 +01:00
Eugen Rochko
ddbe906c25
Fix not updating a status when newer version is fetched manually (#17745) 2022-03-12 09:11:36 +01:00
Claire
24e83246f9 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/models/status.rb`:
  Upstream updated media and edit-related code textually close to glitch-soc
  additions (local-only and content-type).
  Ported upstream changes.
- `app/models/status_edit.rb`:
  Upstream changes textually close to glitch-soc additions (content-type).
  Ported upstream changes.
- `app/serializers/activitypub/note_serializer.rb`:
  Upstream changed how media attachments are handled. Not really a conflict,
  but textually close to glitch-soc additions (directMessage attribute).
  Ported upstream changes.
- `app/services/remove_status_service.rb`:
  Upstream changed how media attachments are handled. Not really a conflict,
  but textually close to glitch-soc additions (DM timeline).
  Ported upstream changes.
- `app/services/update_status_service.rb`:
  Upstream fixed an issue with language selection. Not really a conflict,
  but textually close to glitch-soc additions (content-type).
  Ported upstream changes.
- `db/schema.rb`:
  Upstream added columns to the `status_edits` table, the conflict is because
  of an additional column (`content-type`) in glitch-soc.
  Ported upstream changes.
- `package.json`:
  Upstream dependency (express) textually adjacent to a glitch-soc-specific one
  (favico.js) got updated.
  Updated it as well.
2022-03-10 09:52:45 +01:00
Eugen Rochko
b2cd34474b
Add rate limit for editing (#17728) 2022-03-09 20:06:51 +01:00
Eugen Rochko
d17fb70131
Change how changes to media attachments are stored for edits (#17696)
* Change how changes to media attachments are stored for edits

Fix not being able to re-order media attachments

* Fix not broadcasting updates when polls/media is changed through ActivityPub

* Various fixes and improvements

* Update app/models/report.rb

Co-authored-by: Claire <claire.github-309c@sitedethib.com>

* Add tracking of media attachment description changes

* Change poll in status edit to have a structure closer to the real one

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2022-03-09 09:06:17 +01:00
Eugen Rochko
c0327ff31f
Fix invalid language resulting in no fallback being set on statuses (#17722) 2022-03-09 08:46:05 +01:00
Claire
d9c3109bb6 Merge branch 'main' into glitch-soc/merge-upstream 2022-03-04 01:07:49 +01:00
Eugen Rochko
2ea754b861
Fix duplicate notifications being possible after poll expiration (#17697) 2022-03-04 01:06:33 +01:00
Claire
8743b1ea40 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/views/admin/trends/links/index.html.haml`:
  Not really a conflict, upstream change textually too close to a glitch-soc
  change (removed `javascript_pack_tag` to accomodate for glitch-soc's theming
  system).
  Ported upstream changes.
- `app/views/admin/trends/links/preview_card_providers/index.html.haml`:
  Not really a conflict, upstream change textually too close to a glitch-soc
  change (removed `javascript_pack_tag` to accomodate for glitch-soc's theming
  system).
  Ported upstream changes.
- `app/views/admin/trends/statuses/index.html.haml`:
  Not really a conflict, upstream change textually too close to a glitch-soc
  change (removed `javascript_pack_tag` to accomodate for glitch-soc's theming
  system).
  Ported upstream changes.
- `app/views/admin/trends/tags/index.html.haml`:
  Not really a conflict, upstream change textually too close to a glitch-soc
  change (removed `javascript_pack_tag` to accomodate for glitch-soc's theming
  system).
  Ported upstream changes.
2022-03-02 20:48:27 +01:00
Eugen Rochko
02b8d63fce
Fix report category not being saved in REST API (#17682) 2022-03-02 18:57:08 +01:00
Claire
425a6c90c4 Merge branch 'main' into glitch-soc/merge-upstream 2022-03-02 18:02:48 +01:00
Eugen Rochko
25d3dc4373
Add ability to mark statuses as sensitive from reports in admin UI (#17668)
* Add ability to mark statuses as sensitive from reports in admin UI

* Allow mark as sensitive action on statuses with preview cards
2022-03-01 22:20:29 +01:00
Claire
be493b6c0d Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/models/account.rb`:
  Not a real conflict, just upstream getting rid of unused constants too close
  to glitch-soc-specific contents.
  Removed unused constants like upstream did.
- `app/models/trends.rb`:
  Conflict because glitch-soc disabled email notifications for trending links.
  Upstream has refactored this quite a bit and added trending posts.
  Took upstream code, but disabling the extra trending stuff will come in
  another commit.
- `app/views/admin/trends/links/index.html.haml`:
  Conflict due to glitch-soc's theming system.
  Ported upstream changes accordingly.
2022-02-26 09:29:23 +01:00
Eugen Rochko
27965ce5ed
Add trending statuses (#17431)
* Add trending statuses

* Fix dangling items with stale scores in localized sets

* Various fixes and improvements

- Change approve_all/reject_all to approve_accounts/reject_accounts
- Change Trends::Query methods to not mutate the original query
- Change Trends::Query#skip to offset
- Change follow recommendations to be refreshed in a transaction

* Add tests for trending statuses filtering behaviour

* Fix not applying filtering scope in controller
2022-02-25 00:34:14 +01:00
Claire
55692978ec Merge branch 'main' into glitch-soc/merge-upstream 2022-02-23 19:43:34 +01:00
Claire
1c3e5e44e2
Fix new sign-up notification not working because of incorrect type name (#17629) 2022-02-23 19:30:13 +01:00
Claire
3d25fe0426 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `.github/workflows/build-image.yml`:
  Upstream changed the workflow a bit.
  Conflict comes from us pushing to ghcr while upstream pushes to dockerhub.
  Ported the upstream changes while still pushing to ghcr.
2022-02-23 18:13:12 +01:00
Eugen Rochko
7b816eb5ae
Add notifications for new sign-ups (#16953) 2022-02-23 16:45:22 +01:00
luzpaz
73f5e4a1d9
Fix various typos (#17621)
Found via `codespell -q 3 -S ./CHANGELOG.md,./AUTHORS.md,./config/locales,./app/javascript/mastodon/locales -L ba,keypair,medias,ro`
2022-02-22 20:14:17 +01:00
Claire
f224237862 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `db/schema.rb`:
  Conflict due to glitch-soc adding the `content_type` column on status edits
  and thus having a different schema version number.
  Solved by taking upstream's schema version number, as it is higher than
  glitch-soc's.
2022-02-17 10:58:44 +01:00
Eugen Rochko
564efd0651
Add appeals (#17364)
* Add appeals

* Add ability to reject appeals and ability to browse pending appeals in admin UI

* Add strikes to account page in settings

* Various fixes and improvements

- Add separate notification setting for appeals, separate from reports
- Fix style of links in report/strike header
- Change approving an appeal to not restore statuses (due to federation complexities)
- Change style of successfully appealed strikes on account settings page
- Change account settings page to only show unappealed or recently appealed strikes

* Change appealed_at to overruled_at

* Fix missing method error
2022-02-14 21:27:53 +01:00
Claire
2d3f8055ed Fix attempts at distributing Update activities for local-only posts 2022-02-12 15:41:18 +01:00
Claire
d9379f5331 Merge branch 'main' into glitch-soc/merge-upstream 2022-02-12 11:12:41 +01:00
Eugen Rochko
8f03b7a2fb
Add notifications when a reblogged status has been updated (#17404)
* Add notifications when a reblogged status has been updated

* Change wording to say "edit" instead of "update" and add missing controls

* Replace previous update notifications with the most up-to-date one
2022-02-11 22:20:19 +01:00
Claire
51cad4f7fb Merge branch 'main' into glitch-soc/merge-upstream 2022-02-11 21:53:20 +01:00
Eugen Rochko
c9a52833b6
Fix deletes not being signed in authorized fetch mode (#17484)
Fix #17483
2022-02-11 14:52:45 +01:00
Claire
abd113167b Add ability to change content-type when editing a toot
Content-type defaults to edited toot's content-type to avoid surprising
behaviors when using clients that do not support this feature.
2022-02-10 19:10:14 +01:00
Claire
f1a6f9062e Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/controllers/api/v1/statuses_controller.rb`:
  Upstream moved things around in a place where glitch-soc had support for
  an extra parameter (`content_type`).
  Follow upstream but reintroduce `content_type`.
2022-02-10 19:09:27 +01:00
Claire
63854bee6c
Fix poll votes not being properly reset on poll change (#17498)
* Fix poll votes not being properly reset on poll change

* Fix and add tests

* Fix poll update handling when the number of options changes
2022-02-10 14:26:54 +01:00
Eugen Rochko
1bfcb75105
Fix outdated iso-639 reference in update status service (#17496) 2022-02-10 03:09:44 +01:00
Eugen Rochko
63002cde03
Add editing for published statuses (#17320)
* Add editing for published statuses

* Fix change of multiple-choice boolean in poll not resetting votes

* Remove the ability to update existing media attachments for now
2022-02-10 00:15:30 +01:00
Eugen Rochko
2f8159baad
Add category and rule_ids params to POST /api/v1/reports (#17492) 2022-02-10 00:10:16 +01:00
Claire
692963d43b Merge branch 'main' into glitch-soc/merge-upstream 2022-02-08 18:23:53 +01:00
Eugen Rochko
b6d7726ecb
Remove language detection through cld3 (#17478)
* Remove language detection through cld3

* Update app/helpers/languages_helper.rb

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>
2022-02-08 02:41:17 +01:00
Claire
c8b1e72a4f
Fix compacted JSON-LD possibly causing compatibility issues on forwarding (#17428) 2022-02-03 14:09:04 +01:00
Claire
948235592a
Fix response_to_recipient? CTE (#17427) 2022-02-03 14:07:43 +01:00
Claire
d1ecc323e7
Compact JSON-LD signed incoming activities (#17426)
Co-authored-by: Puck Meerburg <puck@puck.moe>
2022-02-03 14:07:29 +01:00
Claire
a3e0dacf5c Fix response_to_recipient? CTE 2022-02-02 19:55:57 +01:00
Claire
7b969436a0 Fix compacted JSON-LD possibly causing compatibility issues on forwarding 2022-02-02 19:55:57 +01:00
Puck Meerburg
63da32468c Compact JSON-LD signed incoming activities 2022-02-02 16:13:11 +01:00
Claire
94a39f6b68 Fix Sidekiq warning when pushing DMs to direct timeline 2022-01-28 09:07:56 +01:00
Claire
b2915613fb Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `Gemfile.lock`:
  Upstream-updated lib textually too close to glitch-soc-only dep.
  Updated like upstream.
2022-01-28 08:58:32 +01:00
Claire
03d59340da
Fix Sidekiq warnings about JSON serialization (#17381)
* Fix Sidekiq warnings about JSON serialization

This occurs on every symbol argument we pass, and every symbol key in hashes,
because Sidekiq expects strings instead.

See https://github.com/mperham/sidekiq/pull/5071

We do not need to change how workers parse their arguments because this has
not changed and we were already converting to symbols adequately or using
`with_indifferent_access`.

* Set Sidekiq to raise on unsafe arguments in test mode

In order to more easily catch issues that would produce warnings in production
code.
2022-01-28 00:43:56 +01:00
Claire
ad6ddb9bdd Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `config/environments/production.rb`:
  Upstream changed a header but we had different default headers.
  Applied the same change, and also dropped HSTS headers redundant with
  Rails'.
2022-01-26 22:32:21 +01:00
Claire
166cc5b89d
Fix local distribution of edited statuses (#17380)
Because `FanOutOnWriteService#update?` was broken, edits were considered as new
toots and a regular `update` payload was sent.
2022-01-26 20:53:50 +01:00
Eugen Rochko
6505b39e5d
Fix poll updates being saved as status edits (#17373)
Fix #17344
2022-01-26 18:05:39 +01:00
Claire
fe89554a54 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/lib/activitypub/activity/create.rb`:
  Upstream refactored how `Create` activities are handled and how values are
  extracted from `Create`d objects. This conflicted with how glitch-soc
  supported the `directMessage` flag to explicitly distinguish between
  limited and direct messages.
  Ported glitch-soc's changes to latest upstream changes.
- `app/services/fan_out_on_write_service.rb`:
  Upstream largely refactored that file and changed some of the logic.
  This conflicted with glitch-soc's handling of the direct timeline and
  the options to allow replies and boosts in public feeds.
  Ported those glitch-soc changes on top of latest upstream changes.
- `app/services/process_mentions_service.rb`:
  Upstream refactored to move mention-related ActivityPub deliveries to
  `ActivityPub::DeliveryWorker`, while glitch-soc contained an extra check
  to not send local-only toots to remote mentioned users.
  Took upstream's version, as the check is not needed anymore, since it is
  performed at the `ActivityPub::DeliveryWorker` call site already.
- `app/workers/feed_insert_worker.rb`:
  Upstream added support for `update` toot events, while glitch-soc had
  support for an extra timeline support, `direct`.
  Ported upstream changes and extended them to the `direct` timeline.

Additional changes:
- `app/lib/activitypub/parser/status_parser.rb`:
  Added code to handle the `directMessage` flag and take it into account
  to compute visibility.
- `app/lib/feed_manager.rb`:
  Extended upstream's support of `update` toot events to glitch-soc's
  `direct` timeline.
2022-01-19 23:52:48 +01:00
Eugen Rochko
d412a8d1f2
Fix error when processing poll updates (#17333)
Regression from #16697
2022-01-19 22:50:01 +01:00
Eugen Rochko
1060666c58
Add support for editing for published statuses (#16697)
* Add support for editing for published statuses

* Fix references to stripped-out code

* Various fixes and improvements

* Further fixes and improvements

* Fix updates being potentially sent to unauthorized recipients

* Various fixes and improvements

* Fix wrong words in test

* Fix notifying accounts that were tagged but were not in the audience

* Fix mistake
2022-01-19 22:37:27 +01:00
Claire
c42938aed4 Merge branch 'main' into glitch-soc/merge-upstream 2022-01-19 12:32:44 +01:00
Jeong Arm
2d1f082bb6
Fix NameError on ActivityPub::FetchFeaturedCollectionService (#17326)
Related: #16954
2022-01-19 04:08:46 +01:00
Claire
b3bf32a21e Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/views/admin/reports/show.html.haml`:
  Conflicts due to glitch-soc's theming system.
2022-01-17 10:45:25 +01:00
Eugen Rochko
14f436c457
Add notifications for statuses deleted by moderators (#17204) 2022-01-17 09:41:33 +01:00
Claire
2d4b4b0b45 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/controllers/accounts_controller.rb`:
  Upstream introduced support for private pinned toots, but glitch-soc's query
  was a bit different as it filtered out local-only toots.
  Used upstream's query, while adding local-only filtering back.
- `app/controllers/activitypub/collections_controller.rb`:
  Same thing with regards to local-only posts.
- `app/validators/status_pin_validator.rb`:
  Not a real conflict, but the line below was different in glitch-soc due to
  the configurable pinned toots limit.
2022-01-17 08:28:52 +01:00
Claire
d5c9feb7b7
Add support for private pinned posts (#16954)
* Add support for private pinned toots

* Allow local user to pin private toots

* Change wording to avoid "direct message"
2022-01-17 00:49:55 +01:00
Claire
fca50e4b50 Merge branch 'main' into glitch-soc/merge-upstream 2022-01-07 19:51:47 +01:00
Jeong Arm
d3db2eb7fb
Remove custom emojis on domain purge (#17210) 2021-12-30 08:41:09 +01:00
Claire
d4f315f304 Merge branch 'main' into glitch-soc/merge-upstream 2021-12-18 11:44:54 +01:00
Claire
7f803c41e2
Add ability to purge undeliverable domains from admin interface (#16686)
* Add ability to purge undeliverable domains from admin interface

* Add tests
2021-12-17 23:01:21 +01:00
Claire
9b861d56a9 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `.env.production.sample`:
  Copied upstream changes.
- `app/controllers/settings/identity_proofs_controller.rb`:
  Minor conflict due to glitch-soc's extra “enable_keybase” setting.
  Upstream removed keybase support altogether, so did the same.
- `app/controllers/well_known/keybase_proof_config_controller.rb`:
  Minor conflict due to glitch-soc's extra “enable_keybase” setting.
  Upstream removed keybase support altogether, so did the same.
- `lib/mastodon/statuses_cli.rb`:
  Minor conflict due to an optimization that wasn't shared between
  the two versions. Copied upstream's version.
2021-11-26 22:53:55 +01:00
Eugen Rochko
7de0ee7aba
Remove Keybase integration (#17045) 2021-11-26 05:58:18 +01:00
Claire
443ec4f8ba Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/views/admin/tags/index.html.haml`:
  Removed upstream while it had changes in glitch-soc to accomodate for the
  theming system.

Additional changes to accomodate for the theming system:
- `app/views/admin/trends/links/preview_card_providers/index.html.haml`
- `app/views/admin/trends/links/index.html.haml`
- `app/views/admin/trends/tags/index.html.haml`
- `app/views/admin/tags/show.html.haml`
2021-11-25 23:50:35 +01:00
Claire
013bee6afb
Fix filtering DMs from non-followed users (#17042) 2021-11-25 23:46:30 +01:00
Eugen Rochko
6e50134a42
Add trending links (#16917)
* Add trending links

* Add overriding specific links trendability

* Add link type to preview cards and only trend articles

Change trends review notifications from being sent every 5 minutes to being sent every 2 hours

Change threshold from 5 unique accounts to 15 unique accounts

* Fix tests
2021-11-25 13:07:38 +01:00
Claire
2b4c7beb73 Merge branch 'main' into glitch-soc/merge-upstream 2021-11-19 18:51:41 +01:00
Takeshi Umeda
3419d3ec84
Bump chewy from 5.2.0 to 7.2.3 (supports Elasticsearch 7.x) (#16915)
* Bump chewy from 5.2.0 to 7.2.2

* fix style (codeclimate)

* fix style

* fix style

* Bump chewy from 7.2.2 to 7.2.3
2021-11-18 22:02:08 +01:00
Claire
4bb6b1a1e7 Merge branch 'main' into glitch-soc/merge-upstream 2021-11-06 00:15:38 +01:00
Eugen Rochko
39cdf61ab7
Add support for structured data and more OpenGraph tags to link cards (#16938)
Save preview cards under their canonical URL

Increase max redirects to follow from 2 to 3
2021-11-05 23:23:05 +01:00
Claire
e855df149b Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `.env.production.sample`:
  Upstream added new configuration options, uncommented by default.
  Commented them.
- `Gemfile.lock`:
  Upstream updated dependencies textually close to glitch-soc-specific
  dependencies.
  Updated those upstream dependencies.
2021-10-25 20:28:51 +02:00
Claire
ec059317fa
Fix some link previews being incorrectly generated from other prior links (#16885)
* Add tests

* Fix some link previews being incorrectly generated from different prior links

PR #12403 added a cache to avoid redundant queries when the OEmbed endpoint can
be guessed from the URL. This caching mechanism is not perfectly correct as
there is no guarantee that all pages from a given domain share the same
OEmbed provider endpoint.

This PR prevents the FetchOEmbedService from caching OEmbed endpoint that
cannot be generalized by replacing a fully-qualified URL from the endpoint's
parameters, greatly reducing the number of incorrect cached generalizations.
2021-10-21 20:39:35 +02:00
Claire
3f9b28ce26
Add support for fetching Create and Announce activities by URI (#16383)
* Add support for fetching Create and Announce activities by URI

This should improve compatibility with ZAP and offer a way to fetch boosts,
which is currently not possible.

* Add tests
2021-10-21 01:14:04 +02:00
Takeshi Umeda
17f4e457b3
Add remove from followers api (#16864)
* Add followed_by? to account_interactions

* Add RemoveFromFollowersService

* Fix AccountBatch to use RemoveFromFollowersService

* Add remove from followers API
2021-10-18 12:02:35 +02:00
Claire
694c073d1f Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/controllers/admin/dashboard_controller.rb`:
  Upstream completely redesigned the admin dashboard.
  glitch-soc tracked extra features, but that list is
  gone.
  Followed upstram.
- `app/views/admin/dashboard/index.html.haml`
  Upstream completely redesigned the admin dashboard.
  glitch-soc tracked extra features, but that list is
  gone.
  Followed upstram.
2021-10-14 21:44:57 +02:00
Claire
216570ad98
Fix scheduled statuses decreasing statuses counts (#16791)
* Add tests

* Fix scheduled statuses decreasing statuses counts

Fixes #16774
2021-10-14 19:59:21 +02:00
Claire
3622110778 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `streaming/index.js`:
  Filtering code for streaming notifications has been refactored upstream, but
  glitch-soc had similar code for local-only toots in the same places.
  Ported upstream changes, but did not refactor local-only filtering.
2021-09-26 18:28:59 +02:00
Eugen Rochko
a0d4129893
Refactor notifications to go through a separate stream in streaming API (#16765)
Eliminate need to have custom notifications filtering logic in the
streaming API code by publishing notifications into a separate stream
and then simply using the multi-stream capability to subscribe to that
stream when necessary
2021-09-26 13:23:28 +02:00
Claire
4e2a8c9b38 Merge branch 'main' into glitch-soc/merge-upstream 2021-08-20 13:01:50 +02:00
Claire
9ac7e6fef7
Fix remotely-suspended accounts' toots being merged back into timelines (#16628)
* Fix remotely-suspended accounts' toots being merged back into timelines

* Mark remotely-deleted accounts as remotely suspended
2021-08-20 07:40:33 +01:00
Holger
0cae6c07bb
Fix #16603 (#16605)
Fix issue #16603 undefined method `serialize_payload' for Unsuspend Account Service error.
It seems that this service forgot to `include Payloadable` so that `serialize_payload` could not be found in this service.
2021-08-20 07:39:37 +01:00
Claire
8ec4be4233 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `Gemfile.lock`:
  Not a real conflict, upstream-updated dependency (redis) textually too
  close to glitch-soc-only dependecy.
  Updated redis gem like upstream did.
2021-08-09 23:25:49 +02:00
Claire
4ac78e2a06
Add feature to automatically delete old toots (#16529)
* Add account statuses cleanup policy model

* Record last inspected toot to delete to speed up successive calls to statuses_to_delete

* Add service to cleanup a given account's statuses within a budget

* Add worker to go through account policies and delete old toots

* Fix last inspected status id logic

All existing statuses older or equal to last inspected status id must be
kept by the current policy. This is an invariant that must be kept so that
resuming deletion from the last inspected status remains sound.

* Add tests

* Refactor scheduler and add tests

* Add user interface

* Add support for discriminating based on boosts/favs

* Add UI support for min_reblogs and min_favs, rework UI

* Address first round of review comments

* Replace Snowflake#id_at_start with with_random parameter

* Add tests

* Add tests for StatusesCleanupController

* Rework settings page

* Adjust load-avoiding mechanisms

* Please CodeClimate
2021-08-09 23:11:50 +02:00
Claire
763ab0c7eb
Fix owned account notes not being deleted when an account is deleted (#16579)
* Add account_notes relationship

* Add tests

* Fix owned account notes not being deleted when an account is deleted

* Add post-migration to clean up orphaned account notes
2021-08-08 15:29:57 +02:00
Claire
b5e10d07ff Merge branch 'main' into glitch-soc/merge-upstream 2021-07-25 12:48:48 +02:00
Claire
1d67acb72f
Fix scoped order warning in RemoveStatusService (#16531)
Fixes “Scoped order is ignored, it's forced to be batch order.”
2021-07-24 14:41:46 +02:00
Claire
d8629e7b86
Add logging of S3-related errors (#16381) 2021-07-21 18:34:39 +02:00
Claire
5f334807d2 Merge branch 'main' into glitch-soc/merge-upstream 2021-05-22 13:11:31 +02:00
Claire
9a19227f17
Fix some RedisLocks auto-releasing too fast (#16276)
* Fix Delete and Create-related locks expiring too fast

Fixes #16238

By default, RedisLock expires after 10 seconds, which may not be enough to
process statuses, especially when those have attached media files.

This commit extends those 10 seconds to 15 minutes, which should be plenty
enough to handle any status, while being short enough to not waste many
sidekiq job retries in the exceedingly rare case in which a sidekiq process
would crash when processing a `Create` or `Delete`.

* Fix other RedisLock autorelease durations

Fixes #15645

- things that only perform a few simple database queries (e.g. finding and
  saving a record) have been left unchanged, so they'll still use the default
  10s duration
- things that perform significantly more complex database queries have been
  changed to a 5 minutes timeout
- things that perform multiple HTTP queries have been changed to a 15 minutes
  timeout
2021-05-19 23:52:08 +02:00
Claire
dc58d02192 Merge branch 'main' into glitch-soc/merge-upstream 2021-05-12 11:03:05 +02:00
Takeshi Umeda
c403c3695b
Fix to be able to redownload avatar and header (#16190)
* Fix to reset if header and avatar download fails

* Add RedownloadAvatarWorker and RedownloadHeaderWorker
2021-05-11 14:19:22 +02:00
Claire
7ac8bcf209 Merge branch 'main' into glitch-soc/merge-upstream 2021-05-10 21:45:32 +02:00
Claire
afb7882189
Fix blocking someone not clearing up list feeds (#16205) 2021-05-10 17:31:55 +02:00
Claire
87c3a0d0b3 Merge branch 'main' into glitch-soc/merge-upstream 2021-05-09 10:41:28 +02:00
Takeshi Umeda
9da5e0b350
Fix webfinger_update_due to run WebFinger on stale activitypub-account (#16182) 2021-05-08 16:22:18 +02:00
Claire
50b430d9a2 Merge branch 'main' into glitch-soc/merge-upstream
- `app/views/statuses/_simple_status.html.haml`:
  Small markup change in glitch-soc, on a line that has been modified by
  upstream. Ported upstream changes.
2021-05-07 18:21:59 +02:00
Eugen Rochko
74081433d0
Change trending hashtags to be affected be reblogs (#16164)
If a status with a hashtag becomes very popular, it stands to
reason that the hashtag should have a chance at trending

Fix no stats being recorded for hashtags that are not allowed
to trend, and stop ignoring bots

Remove references to hashtags in profile directory from the code
and the admin UI
2021-05-07 14:33:43 +02:00
Eugen Rochko
2c77d97e0d
Add joined date to profiles in web UI (#16169) 2021-05-07 14:33:19 +02:00
Eugen Rochko
fab65848d2
Fix empty home feed before first follow has finished processing (#16152)
Change queue of merge worker from pull to default
2021-05-04 04:45:08 +02:00
Eugen Rochko
f627d2eb93
Fix trying to fetch key from empty URI when verifying HTTP signature (#16100) 2021-05-01 23:18:59 +02:00
Eugen Rochko
daccc07dc1
Change auto-following admin-selected accounts, show in recommendations (#16078) 2021-04-24 17:01:43 +02:00
Claire
536892b8ae Merge branch 'main' into glitch-soc/merge-upstream 2021-04-21 13:52:03 +02:00
Claire
0b36e3419d
Fix processing of remote Delete activities (#16084)
* Add tests

* Ensure deleted statuses are marked as such

* Save some redis memory by not storing URIs in delete_upon_arrival values

* Avoid possible race condition when processing incoming Deletes

* Avoid potential duplicate Delete forwards

* Lower lock durations to reduce issues in case of hard crash of the Rails process

* Check for `lock.aquired?` and improve comment

* Refactor RedisLock usage in app/lib/activitypub

* Fix using incorrect or non-existent sender for relaying Deletes
2021-04-21 04:46:09 +02:00
Claire
e2a2bc9021 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `README.md`:
  Upstream updated copyright year, we don't mention it so kept our version.
- `app/controllers/admin/dashboard_controller.rb`:
  Not really a conflict, upstream change (removing the spam checker) too close
  to glitch-soc changes. Ported upstream changes.
- `app/models/form/admin_settings.rb`:
  Same.
- `app/services/remove_status_service.rb`:
  Same.
- `app/views/admin/settings/edit.html.haml`:
  Same.
- `config/settings.yml`:
  Same.
- `config/environments/production.rb`:
  Not a real conflict, upstream added a default HTTP header, but we have
  extra headers in glitch-soc.
  Added the header.
2021-04-20 12:17:14 +02:00
Eugen Rochko
ca3bc1b09f
Refactor StatusReachFinder to handle followers and relays as well (#16051) 2021-04-17 15:41:57 +02:00
Eugen Rochko
6d6000f61f
Fix remote reporters not receiving suspend/unsuspend activities (#16050) 2021-04-17 14:55:46 +02:00
Eugen Rochko
dde8739020
Fix reports of already suspended accounts being recorded (#16047) 2021-04-16 22:01:05 +02:00
Eugen Rochko
619fad6cf8
Remove spam check and dependency on nilsimsa gem (#16011) 2021-04-11 11:22:50 +02:00
Claire
f60c99a8fb Merge branch 'main' into glitch-soc/merge-upstream 2021-03-24 10:53:50 +01:00
Claire
cbd0ee1d07
Update Mastodon to Rails 6.1 (#15910)
* Update devise-two-factor to unreleased fork for Rails 6 support

Update tests to match new `rotp` version.

* Update nsa gem to unreleased fork for Rails 6 support

* Update rails to 6.1.3 and rails-i18n to 6.0

* Update to unreleased fork of pluck_each for Ruby 6 support

* Run "rails app:update"

* Add missing ActiveStorage config file

* Use config.ssl_options instead of removed ApplicationController#force_ssl

Disabled force_ssl-related tests as they do not seem to be easily testable
anymore.

* Fix nonce directives by removing Rails 5 specific monkey-patching

* Fix fixture_file_upload deprecation warning

* Fix yield-based test failing with Rails 6

* Use Rails 6's index_with when possible

* Use ActiveRecord::Cache::Store#delete_multi from Rails 6

This will yield better performances when deleting an account

* Disable Rails 6.1's automatic preload link headers

Since Rails 6.1, ActionView adds preload links for javascript files
in the Links header per default.

In our case, that will bloat headers too much and potentially cause
issues with reverse proxies. Furhermore, we don't need those links,
as we already output them as HTML link tags.

* Switch to Rails 6.0 default config

* Switch to Rails 6.1 default config

* Do not include autoload paths in the load path
2021-03-24 10:44:31 +01:00
Claire
1b6a21e6bc Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/validators/status_length_validator.rb`:
  Conflict due to glitch-soc's configurable maximum toot chars.
  Ported upstream changes.
2021-03-11 16:08:15 +01:00
Claire
5614e6724e
Fix URL scanning in note length validator and preview card fetching (#15827)
* Add tests

* Fix URL scanning in note length validator and preview card fetching
2021-03-04 00:12:26 +01:00
Claire
d8fdbb054e Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/validators/status_length_validator.rb`:
  Upstream changes too close to glitch-soc MAX_CHARS changes, but not a real
  conflict.
  Applied upstream changes.
- `package.json`:
  glitch-soc-only dependency textually too close to a dependency updated
  upstream, not a real conflict.
  Applied upstream changes.
2021-03-02 12:06:58 +01:00
Claire
65db262550
Update twitter-text from 1.14 to 3.1.0 and fix toot character counting (#15382)
* Update twitter-text from 1.14 to 3.1.0

* Disable emoji parsing

* Properly depend on twitter-text for url detection

* Fix some URLs being wrongly detected client-side

* Add test for server-side validation of non-autolinkable URLs

* Fix server-side status length counting
2021-03-02 12:02:56 +01:00
Claire
5f4c0b79c2
Change ResolveAccountService's handling of skip_webfinger (#15750)
* Change ResolveAccountService's handling of skip_webfinger

Change it so it never makes any webfinger query, as the name would imply.

* Add tests

* Change FollowService to not take an URI for target_account

* Restore domain-block check in FollowService

* Fix tests
2021-02-24 06:32:13 +01:00
Claire
5e11f3a6e1 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/javascript/styles/mastodon/modal.scss`:
  For some reason we changed the file loading path in glitch-soc,
  but now upstream has completely changed how the logo is loaded.
  Applied upstream changes.
2021-02-12 10:28:32 +01:00
Eugen Rochko
f8972d4503
Fix YouTube embeds failing due to YouTube serving wrong OEmbed URLs (#15716) 2021-02-12 05:45:38 +01:00
Claire
b81710c02c Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `CONTRIBUTING.md`:
  Not a real conflict, glitch-soc quotes the upstream file, which has been
  changed. Update the quote.
2021-01-26 14:01:30 +01:00
Takeshi Umeda
7f1c56954b
Fix first return value of FetchLinkCardService.html method (#15630) 2021-01-25 09:22:41 +01:00
luigi
7ea9588520
Use Enumerable#filter_map in more places (#15527) 2021-01-22 16:28:15 +01:00
luigi
eb51e43fb4
Optimize some regex matching (#15528)
* Use Regex#match?

* Replace =~ too

* Avoid to call match? from Nil

* Keep value of Regexp.last_match
2021-01-22 10:09:08 +01:00
Claire
33d30632fb Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/models/public_feed.rb`:
  Upstream refactored a bit, glitch-soc had specific code for local-only
  statuses.
  Updated glitch-soc's specific code accordingly.
2021-01-11 11:55:42 +01:00
luigi
087ed84367
Optimize map { ... }.compact calls (#15513)
* Optimize map { ... }.compact

using Enumerable#filter_map, supported since Ruby 2.7

* Add poyfill for Enumerable#filter_map
2021-01-10 00:32:01 +01:00
abcang
efffdd3778
Fix rubocop config and warnings (#15503)
* disable NewCops

* update TargetRubyVersion

* Fix Lint/MissingSuper for ActiveModelSerializers::Model

* Fix Lint/MissingSuper for feed

* Fix Lint/FloatComparison

* Do not use instance variables
2021-01-07 09:40:55 +01:00
Claire
90528f43bc Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `config/webpack/configuration.js`:
  Upstream updated the `js-yaml` dependency, which changed how to call it.
  Those changes conflicted because that code is pretty different in glitch-soc
  which has to deal with its more complex theming system.
  Proceeded to the same compatibility changes in glitch-soc's code.
- `package.json` and `yarn.lock`:
  Not really a conflict, just glitch-soc-specific dependencies textually too
  close to some dependencies updated upstream.
2021-01-06 18:08:06 +01:00
ThibG
f1f96ebf02
Fix being able to import more than allowed number of follows (#15384)
* Fix being able to import more than allowed number of follows

Without this commit, if someone tries importing a second list of accounts to
follow before the first one has been processed, this will queue imports for
the two whole lists, even if they exceed the account's allowed number of
outgoing follows.

This commit changes it so the individual queued imports aren't exempt from
the follow limit check (they remain exempt from the rate-limiting check
though).

* Catch validation errors to not re-queue failed follows

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-26 23:52:46 +01:00
Eugen Rochko
e89648574f
Fix error when changing ACL on missing objects during suspension (#15420) 2020-12-24 15:53:45 +01:00
Eugen Rochko
ba0b79fc5c
Fix undefined method error when batch-removing statuses with ElasticSearch on (#15421) 2020-12-24 15:53:26 +01:00
ThibG
de57efd055
Fix mentions not being deleted efficiently (#15416)
As a regression from the recent optimizations, mentions were left untouched
until `account.destroy`, which would then delete them individually,
and executing queries to find and delete associated notifications, resulting
in a massive slowdown.

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-23 16:43:11 +01:00
Eugen Rochko
473abc35a1
Fix trying to privatize empty media attachments (#15414) 2020-12-23 15:47:03 +09:00
Claire
537afa00f3 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/lib/feed_manager.rb`:
  Not a real conflict, glitch-soc-only DM-related method
  too close to changed upstream stuff.
  Ported upstream changes.
- `app/services/batched_remove_status_service.rb`:
  Additional logic in glitch-soc to clear DMs from timelines.
  Ported upstream changes and fixed the DM TL clearing logic.
- `app/workers/scheduler/feed_cleanup_scheduler.rb`:
  Additional code in glitch-soc to clear DM timelines.
  Ported upstream changes.
2020-12-23 01:47:45 +01:00
ThibG
814b7775fb
Improve performances of deleting favourites when deleting accounts (#15412)
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-23 01:35:02 +01:00
ThibG
2ab3e91eaf
Fix BatchedRemoveStatusService not working without ES in rails console (#15408)
Not a huge deal but may cause surprising failures in custom scripts and
development.

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-23 00:04:52 +01:00
ThibG
3249d35bdc
Improve account deletion performances further (#15407)
* Delete status records by batches of 50

* Do not precompute values that are only used once

* Do not generate redis events for removal of public toots older than two weeks

* Filter reported toots a priori for polls and status deletion

* Do not process reblogs when cleaning up public timelines

As in Mastodon proper, reblogs don't appear in public TLs

* Clean the deleted account's own feed in one go

* Refactor Account#clean_feed_manager and List#clean_feed_manager

* Delete instead of destroy a few more associations

* Fix preloading

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-22 23:57:46 +01:00
ThibG
f18349640b
Fix batch order warnings in BatchedRemoveStatusService (#15409)
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-22 20:24:55 +01:00
Eugen Rochko
9915d11c0d
Fix unnecessary queries when batch-removing statuses, 100x faster (#15387) 2020-12-22 17:13:55 +01:00
Claire
38bfaf8855 Merge branch 'master' into glitch-soc/merge-upstream 2020-12-21 18:28:23 +01:00
ThibG
43961035a9
Fix some notifications not being deleted on poll/status deletion (#15402)
* Fix deleting polls not deleting notifications

* Fix fav notification deletion when deleting a toot

* Refactor DeleteAccountService spec

* Add DeleteAccountService tests for other associations and notifications

* Add favourite handling spec in status removal

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-21 18:22:17 +01:00
ThibG
7bf3c6e57b
Fix AccountDeletionWorker crashing and clogging sidekiq queues (#15380)
* Fix account deletion workers being queued multiple times for a single account

* Fix poll votes being unnecessarily instantiated on poll deletion

* Fix favourites being unnecessarily instantiated on status deletion

* Remove inaccurate comments

* Delete polls instead of destroying them

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-20 18:25:00 +01:00
Claire
f9d000ebae Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/services/resolve_url_service.rb`:
  The private toot search by URL hack has been revamped upstream.
  Took upstream's version.
2020-12-19 00:55:12 +01:00
ThibG
a60d9335d8
Fix resolving accounts sometimes creating duplicate records for a given AP id (#15364)
* Fix ResolveAccountService accepting mismatching acct: URI

* Set attributes that should be updated regardless of suspension

* Fix key fetching

* Automatically merge remote accounts with duplicate `uri`

* Add tests

* Add "tootctl accounts fix-duplicates"

Finds duplicate accounts sharing a same ActivityPub `id`, re-fetch them and
merge them under the canonical `acct:` URI.

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-18 23:26:26 +01:00
Eugen Rochko
eb35be0431
Fix follow limit preventing re-following of a moved account (#14207) 2020-12-18 09:18:31 +01:00
ThibG
b1feb47055
Improve searching for private toots from URL (#14856)
* Improve searching for private toots from URL

Most of the time, when sharing toots, people use the toot URL rather than
the toot URI, which makes sense since it is the user-facing URL.

In Mastodon's case, the URL and URI are different, and Mastodon does not
have an index on URL, which means searching a private toot by URL is done
with a slow query that will only succeed for very recent toots.

This change gets rid of the slow query, and attempts to guess the URI from
URL instead, as Mastodon's are predictable.

* Add tests

* Only return status with guessed uri if url matches

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-17 06:51:49 +01:00
Claire
e4f8679eae Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/models/form/admin_settings.rb`:
  New setting added upstream. Ported it.
- `app/views/statuses/_simple_status.html.haml`:
  Upstream removed RTL classes. Did the same.
- `config/settings.yml`:
  New setting added upstream. Ported it.
2020-12-15 14:27:06 +01:00
ThibG
1390cc194b
Add indication to admin UI of whether a report has been forwarded (#13237)
* Add indication to admin UI of whether a report has been forwarded

* Rework how forwarded status is displayed

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-15 04:30:15 +01:00
Claire
1978f7265e Fix processing of Delete activities on direct messages 2020-12-14 23:05:32 +01:00
Claire
f8d867bac4 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/services/remove_status_service.rb`:
  Conflict caused by us having a distinc Direct timeline.
  Ported upstream changes.
- `app/javascript/mastodon/features/compose/components/compose_form.js`:
  Conflict between glitch-soc's variable character limit and upstream
  refactoring that part of the code.
  Ported upstream changes.
2020-12-05 17:33:37 +01:00
Eugen Rochko
e7e099d1a0
Fix deletes not reaching every server that interacted with status (#15200)
Extract logic for determining ActivityPub inboxes to send deletes
to to its own class and explicitly include the person the status
replied to (even if not mentioned), people who favourited it, and
people who replied to it (though that one is still not recursive)
2020-11-27 15:48:31 +01:00
Thibaut Girka
e71b16ae2b Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `Gemfile.lock`:
  glitch-soc-only dependency textually too close to updated upstream
  dependencies. Updated to upsteam dependencies.
2020-11-25 15:31:45 +01:00
ThibG
66832cbf33
Undo outgoing follows when suspending a remote account (#15188) 2020-11-21 21:27:00 +01:00
Thibaut Girka
48f0f3ffee Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `.github/ISSUE_TEMPLATE/bug_report.md`:
  Upstream added the `bug` label to bug reports.
  Did the same.
- `app/services/fan_out_on_write_service.rb`:
  Upstream put DMs back into timelines, glitch-soc was already doing it.
  Ignored upstream changes.
2020-11-20 13:27:48 +01:00
ThibG
8b8004a962
Fix webfinger redirect handling in ResolveAccountService (#15187)
* Fix webfinger redirect handling in ResolveAccountService

ResolveAccountService#process_webfinger! handled a one-step webfinger
redirection, but only accepting the result if it matched the exact URI passed
as input, defeating the point of a redirection check.

Instead, use the same logic as in `ActivityPub::FetchRemoteAccountService`,
updating the resulting `acct:` URI with the result of the first webfinger
query.

* Add tests
2020-11-19 19:52:06 +01:00
ThibG
96c1e71329
Add import/export feature for bookmarks (#14956)
* Add ability to export bookmarks

* Add support for importing bookmarks

* Add bookmark import tests

* Add bookmarks export test
2020-11-19 17:48:13 +01:00
ThibG
2f6831f318
Fix sending spurious Rejects when processing remote account deletion (#15104)
* Fix sending spurious Rejects when processing remote account deletion

* Make skip_side_effects imply skip_activitypub
2020-11-19 17:39:47 +01:00
Eugen Rochko
df1653174b
Add cache buster feature for media files (#15155)
Nginx can be configured to bypass proxy cache when a special header
is in the request. If the response is cacheable, it will replace
the cache for that request. Proxy caching of media files is
desirable when using object storage as a way of minimizing bandwidth
costs, but has the drawback of leaving deleted media files for
a configured amount of cache time. A cache buster can make those
media files immediately unavailable. This especially makes sense
when suspending and unsuspending an account.
2020-11-19 17:38:06 +01:00
ThibG
966593393e
Fix DMs not appearing into timelines (#15182)
Fixes #15179
2020-11-19 00:23:46 +01:00
Thibaut Girka
10bd6f415d Improve searching for private toots from URL
Most of the time, when sharing toots, people use the toot URL rather than
the toot URI, which makes sense since it is the user-facing URL.

In Mastodon's case, the URL and URI are different, and Mastodon does not
have an index on URL, which means searching a private toot by URL is done
with a slow query that will only succeed for very recent toots.

This change gets rid of the slow query, and attempts to guess the URI from
URL instead, as Mastodon's are predictable.
2020-11-12 23:44:30 +01:00
Thibaut Girka
ecd83f495c Merge branch 'master' into glitch-soc/merge-upstream 2020-11-09 10:53:53 +01:00
ThibG
1e64666662
Fix crash in SuspendAccountWorker (#15106)
* Fix crash in SuspendAccountWorker

`follows` is an array thanks to `to_a`

* Fix code style issue

Co-authored-by: Eugen Rochko <eugen@zeonfederated.com>
2020-11-08 18:29:48 +01:00
Thibaut Girka
0437d70628 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/controllers/follower_accounts_controller.rb`:
  Conflict due to upstream changing suspension logic while
  glitch-soc has an extra option to hide followers count.
  Ported upstream changes.
2020-11-08 14:20:35 +01:00
Eugen Rochko
3134691948
Add support for reversible suspensions through ActivityPub (#14989) 2020-11-08 00:28:39 +01:00
Thibaut Girka
412218af2e Merge branch 'master' into glitch-soc/merge-upstream 2020-11-07 18:23:02 +01:00
ThibG
ee8cf246cf
Fix crashes in SuspendAccountService/UnsuspendAccountService (#15100)
* Fix crashes in SuspendAccountService/UnsuspendAccountService

* Catch filesystem errors
2020-11-07 13:16:54 +01:00
ThibG
f84239ecab
Fix suspension/unsuspension not working because of FeedManager change (#15099) 2020-11-07 13:16:00 +01:00
Thibaut Girka
9c34090db8 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/services/process_mentions_service.rb`:
  glitch-soc local-only handling, ported upstream changes
2020-10-26 15:48:16 +01:00
ThibG
1f945e7390
Fix followers synchronization mechanism not being triggered on mentions (#15026)
e.g. if someone on an instance that previously had followers gets mentioned
in a private toot, before this PR, they would not receive a
Collection-Synchronization header and may show the toot to the former followers
in addition to the mentioned person.
2020-10-23 14:22:16 +02:00
ThibG
fb586584f2
Fix account processing failing because of large collections (#15027)
Fixes #15025
2020-10-23 14:21:31 +02:00
Thibaut Girka
ec49aa8175 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `.github/dependabot.yml`:
  Updated upstream, we deleted it to not be flooded by Depandabot.
  Kept deleted.
- `Gemfile.lock`:
  Puma updated on both sides, went for the most recent version.
- `app/controllers/api/v1/mutes_controller.rb`:
  Upstream updated the serializer to support timed mutes, while
  glitch-soc added a custom API ages ago to get information that
  is already available elsewhere.
  Dropped the glitch-soc-specific API, went with upstream changes.
- `app/javascript/core/admin.js`:
  Conflict due to changing how assets are loaded. Went with upstream.
- `app/javascript/packs/public.js`:
  Conflict due to changing how assets are loaded. Went with upstream.
- `app/models/mute.rb`:
  🤷
- `app/models/user.rb`:
  New user setting added upstream while we have glitch-soc-specific
  user settings. Added upstream's user setting.
- `config/settings.yml`:
  Upstream added a new user setting close to a user setting we had
  changed the defaults for. Added the new upstream setting.
- `package.json`:
  Upstream dependency updated “too close” to a glitch-soc-specific
  dependency. No real conflict. Updated the dependency.
2020-10-21 19:10:50 +02:00
ThibG
ca56527140
Add follower synchronization mechanism (#14510)
* Add support for followers synchronization on the receiving end

Check the `collectionSynchronization` attribute on `Create` and `Announce`
activities and synchronize followers from provided collection if possible.

* Add tests for followers synchronization on the receiving end

* Add support for follower synchronization on the sender's end

* Add tests for the sending end

* Switch from AS attributes to HTTP header

Replace the custom `collectionSynchronization` ActivityStreams attribute by
an HTTP header (`X-AS-Collection-Synchronization`) with the same syntax as
the `Signature` header and the following fields:
- `collectionId` to specify which collection to synchronize
- `digest` for the SHA256 hex-digest of the list of followers known on the
   receiving instance (where “receiving instance” is determined by accounts
   sharing the same host name for their ActivityPub actor `id`)
- `url` of a collection that should be fetched by the instance actor

Internally, move away from the webfinger-based `domain` attribute and use
account `uri` prefix to group accounts.

* Add environment variable to disable followers synchronization

Since the whole mechanism relies on some new preconditions that, in some
extremely rare cases, might not be met, add an environment variable
(DISABLE_FOLLOWERS_SYNCHRONIZATION) to disable the mechanism altogether and
avoid followers being incorrectly removed.

The current conditions are:
1. all managed accounts' actor `id` and inbox URL have the same URI scheme and
   netloc.
2. all accounts whose actor `id` or inbox URL share the same URI scheme and
   netloc as a managed account must be managed by the same Mastodon instance
   as well.

As far as Mastodon is concerned, breaking those preconditions require extensive
configuration changes in the reverse proxy and might also cause other issues.

Therefore, this environment variable provides a way out for people with highly
unusual configurations, and can be safely ignored for the overwhelming majority
of Mastodon administrators.

* Only set follower synchronization header on non-public statuses

This is to avoid unnecessary computations and allow Follow-related
activities to be handled by the usual codepath instead of going through
the synchronization mechanism (otherwise, any Follow/Undo/Accept activity
would trigger the synchronization mechanism even if processing the activity
itself would be enough to re-introduce synchronization)

* Change how ActivityPub::SynchronizeFollowersService handles follow requests

If the remote lists a local follower which we only know has sent a follow
request, consider the follow request as accepted instead of sending an Undo.

* Integrate review feeback

- rename X-AS-Collection-Synchronization to Collection-Synchronization
- various minor refactoring and code style changes

* Only select required fields when computing followers_hash

* Use actor URI rather than webfinger domain in synchronization endpoint

* Change hash computation to be a XOR of individual hashes

Makes it much easier to be memory-efficient, and avoid sorting discrepancy issues.

* Marginally improve followers_hash computation speed

* Further improve hash computation performances by using pluck_each
2020-10-21 18:04:09 +02:00
OSAMU SATO
96761752ec
Add duration parameter to muting. (#13831)
* Adding duration to muting.

* Remove useless checks
2020-10-13 01:01:14 +02:00
Eugen Rochko
5e1364c448
Add IP-based rules (#14963) 2020-10-12 16:33:49 +02:00
Eugen Rochko
7d985f2aac
Remove dependency on goldfinger gem (#14919)
There are edge cases where requests to certain hosts timeout when
using the vanilla HTTP.rb gem, which the goldfinger gem uses. Now
that we no longer need to support OStatus servers, webfinger logic
is so simple that there is no point encapsulating it in a gem, so
we can just use our own Request class. With that, we benefit from
more robust timeout code and IPv4/IPv6 resolution.

Fix #14091
2020-10-08 00:34:57 +02:00
Thibaut Girka
a7aedebc31 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `Gemfile.lock`:
  Not a real conflict, upstream updated dependencies that were too close to
  glitch-soc-only ones in the file.
- `app/controllers/oauth/authorized_applications_controller.rb`:
  Upstream changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc's theming system.
  Ported upstream changes.
- `app/controllers/settings/base_controller.rb`:
  Upstream refactored and changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc's theming system.
  Ported upstream changes.
- `app/controllers/settings/sessions_controller.rb`:
  Upstream refactored and changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc's theming system.
  Ported upstream changes.
- `app/models/user.rb`:
  Upstream refactored and changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc not preventing moved accounts from logging
  in.
  Ported upstream changes while keeping the ability for moved accounts to log
  in.
- `app/policies/status_policy.rb`:
  Upstream refactored and changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc's local-only toots.
  Ported upstream changes.
- `app/serializers/rest/account_serializer.rb`:
  Upstream refactored and changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc's ability  to hide followers count.
  Ported upstream changes.
- `app/services/process_mentions_service.rb`:
  Upstream refactored and changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc's local-only toots.
  Ported upstream changes.
- `package.json`:
  Not a real conflict, upstream updated dependencies that were too close to
  glitch-soc-only ones in the file.
2020-09-28 14:13:30 +02:00
Eugen Rochko
974b1b79ce
Add option to be notified when a followed user posts (#13546)
* Add bell button

Fix #4890

* Remove duplicate type from post-deployment migration

* Fix legacy class type mappings

* Improve query performance with better index

* Fix validation

* Remove redundant index from notifications
2020-09-18 17:26:45 +02:00
Eugen Rochko
ed099d8bdc
Change account suspensions to be reversible by default (#14726) 2020-09-15 14:37:58 +02:00
Thibaut Girka
9748f074a3 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/controllers/api/v1/timelines/public_controller.rb
- app/lib/feed_manager.rb
- app/models/status.rb
- app/services/precompute_feed_service.rb
- app/workers/feed_insert_worker.rb
- spec/models/status_spec.rb

All conflicts are due to upstream refactoring feed management and us having
local-only toots on top of that. Rewrote local-only toots management for
upstream's changes.
2020-09-08 16:26:47 +02:00
Eugen Rochko
65760f59df
Refactor feed manager (#14761) 2020-09-08 03:41:16 +02:00
Eugen Rochko
e8bc187845
Refactor how public and tag timelines are queried (#14728) 2020-09-07 11:02:04 +02:00
Thibaut Girka
3fefb7909b Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `config/webpack/shared.js`:
  Upstream has changed how Tesseract.js gets included and dropped a dependency.
  The conflict is caused by glitch-soc having different code due to its
  theming system.
  Ported upstream changes.
- `lib/mastodon/version.rb`:
  Upstream refactor/code style change in a place we replaced upstream's
  repo URL with ours.
  Ported upstram changes, keeping our repo URL.
- `yarn.lock`:
  Upstream dropped dependencies, one of which was textually too close to
  a glitch-soc-specific dependency. Not a real conflict.
2020-09-01 17:02:18 +02:00
Eugen Rochko
1c308af84c
Change own direct-visibility statuses to be in the home feed again (#14711)
And remove highlighting in web UI

Full circle from #8940
2020-09-01 13:30:42 +02:00
dependabot[bot]
8972e5f7f6
Bump rubocop from 0.86.0 to 0.88.0 (#14412)
* Bump rubocop from 0.86.0 to 0.88.0

Bumps [rubocop](https://github.com/rubocop-hq/rubocop) from 0.86.0 to 0.88.0.
- [Release notes](https://github.com/rubocop-hq/rubocop/releases)
- [Changelog](https://github.com/rubocop-hq/rubocop/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rubocop-hq/rubocop/compare/v0.86.0...v0.88.0)

Signed-off-by: dependabot[bot] <support@github.com>

* Fix for latest RuboCop

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>
2020-09-01 03:04:00 +02:00
Takeshi Umeda
b1bb1906be
Fix limited follower id in fan-out-on-write service (#14709) 2020-08-31 18:11:27 +02:00
Thibaut Girka
8c3c27bf06 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/controllers/accounts_controller.rb`:
  Upstream change too close to a glitch-soc change related to
  instance-local toots. Merged upstream changes.
- `app/services/fan_out_on_write_service.rb`:
  Minor conflict due to glitch-soc's handling of Direct Messages,
  merged upstream changes.
- `yarn.lock`:
  Not really a conflict, caused by glitch-soc-only dependencies
  being textually too close to updated upstream dependencies.
  Merged upstream changes.
2020-08-30 16:13:08 +02:00
Eugen Rochko
ce8f33dd2f
Fix inefficiencies in fan-out-on-write service (#14682) 2020-08-30 12:33:59 +02:00
Akihiko Odaki
41eeb9ebaa
Use Status.group instead of Status.distinct in HashQueryService (#14662)
DISTINCT clause removes duplicated records according to all the selected
attributes. In reality, it can remove duplicated records only looking at
statuses.id, but the clause confuses the query planner and yields
insufficient performance.
The behavior is also problematic if the scope produced by HashQueryService
is used to query columns without id (using pluck method, for example). The
scope is expected to contain unique statuses, but the uniquness will be
evaluated with some arbitrary columns other than id.

GROUP BY clause resolves those problem by explicitly specifying the
column to take into account for the record distinction.

A workaround for the problem of DISTINCT clause in
Api::V1::Timelines::TagController is no longer necessary and removed.
2020-08-25 13:39:35 +02:00
Thibaut Girka
2cfa142961 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `yarn.lock`:
  No real conflict, just a glitch-soc-only dependency textually too close to
  one updated upstream. Ported upstream changes.
2020-08-01 20:24:23 +02:00
Takeshi Umeda
156af70e4d
Fix following restriction not working when exact match in account search (#14394) 2020-07-26 23:58:36 +02:00
Thibaut Girka
4d950c71e9 Merge branch 'master' into glitch-soc/master 2020-07-24 11:26:40 +02:00
Takeshi Umeda
939a47393c
Fix duplicate notification and delivery of mentions (#14378) 2020-07-23 15:41:49 +02:00
Thibaut Girka
b9f351d845 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/javascript/mastodon/actions/compose.js`:
  Not a “real” conflict, but change too close to a change we made to
  fix the vanilla WebUI locally pushing authored local-only toots in the
  public TL view.
2020-07-22 13:38:17 +02:00
Takeshi Umeda
7540e235a2
Fix movie width and frame_rate returning nil (#14357)
* Fix movie width and frame_rate returning nil

* Add StreamValidationError and raise

* Fix code style
2020-07-19 22:28:27 +02:00
Thibaut Girka
1c6c40d17c Merge branch 'master' into glitch-soc/merge-upstream 2020-07-16 14:43:50 +02:00
ThibG
d658af7ff8
Fix removing allowed domains being done synchronously (#14302)
* Fix removing allowed domains being done synchronously

* Add tests
2020-07-15 21:08:19 +02:00
Thibaut Girka
3f60b096b5 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `config/routes.rb`:
  Upstream disabled E2EE routes, which we did earlier, but slightly
  differently. Took upstream's version.
2020-07-15 15:32:40 +02:00
Eugen Rochko
98b3b80d6b
Fix account domain block not properly unfollowing accounts from domain (#14304)
Fix #14303
2020-07-13 19:33:10 +02:00
ThibG
ee5a4030f7
Fix removing a DomainAllow rule wiping known accounts in open federation mode (#14298)
Fixes #14296
2020-07-12 22:11:49 +02:00
Thibaut Girka
e9ad99bc93 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `package.json`:
  Not really a conflict, just some glitch-soc-specific dependency
  too close to an upstream-updated one.
2020-07-07 15:58:45 +02:00
Eugen Rochko
ed04697510
Change User-Agent of link preview fetching service to include "Bot" (#14248)
This forces Twitter to render OpenGraph tags in the response
2020-07-07 10:55:18 +02:00
Thibaut Girka
9c3b9d5ab9 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `.env.production.sample`:
  Upstream deleted it, I decided to keep it.
- `package.json`:
  Upstream updated a dependency textually too close to wavesurfer.js
  which has been deleted from upstream but is kept in glitch-soc for now.
2020-06-30 23:00:20 +02:00
Eugen Rochko
64aac30733
Add customizable thumbnails for audio and video attachments (#14145)
- Change audio files to not be stripped of metadata
- Automatically extract cover art from audio if it exists
- Add `thumbnail` parameter to `POST /api/v1/media`, `POST /api/v2/media` and `PUT /api/v1/media/:id`
- Add `icon` to represent it in attachments in ActivityPub
- Fix `preview_url` containing URL of missing missing image when there is no thumbnail instead of null
- Fix duration of audio not being displayed on public pages until the file is loaded
2020-06-29 13:56:55 +02:00
Thibaut Girka
12c8ac9e14 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/controllers/activitypub/collections_controller.rb`:
  Conflict due to glitch-soc having to take care of local-only
  pinned toots in that controller.
  Took upstream's changes and restored the local-only special
  handling.
- `app/controllers/auth/sessions_controller.rb`:
  Minor conflicts due to the theming system, applied upstream
  changes, adapted the following two files for glitch-soc's
  theming system:
  - `app/controllers/concerns/sign_in_token_authentication_concern.rb`
  - `app/controllers/concerns/two_factor_authentication_concern.rb`
- `app/services/backup_service.rb`:
  Minor conflict due to glitch-soc having to handle local-only
  toots specially. Applied upstream changes and restored
  the local-only special handling.
- `app/views/admin/custom_emojis/index.html.haml`:
  Minor conflict due to the theming system.
- `package.json`:
  Upstream dependency updated, too close to a glitch-soc-only
  dependency in the file.
- `yarn.lock`:
  Upstream dependency updated, too close to a glitch-soc-only
  dependency in the file.
2020-06-09 10:39:20 +02:00
ThibG
89f40b6c3e
Make domain block/silence/reject-media code more robust (#13424)
* Split media cleanup from reject-media domain blocks to its own service

* Slightly improve ClearDomainMediaService error handling

* Lower DomainClearMediaWorker to lowest-priority queue

* Do not catch ActiveRecord::RecordNotFound in domain block workers

* Fix DomainBlockWorker spec labels

* Add some specs

* Change domain blocks to immediately mark accounts as suspended

Rather than doing so sequentially, account after account, while cleaning
their data. This doesn't change much about the time the block takes to
complete, but it immediately prevents interaction with the blocked domain,
while up to now, it would only be guaranteed when the process ends.
2020-06-09 10:32:00 +02:00
Takeshi Umeda
d890abfcab
Fix performance of follow import (#13836) 2020-06-09 10:26:58 +02:00
Takeshi Umeda
643065799b
Fix activity not being signed (#13948) 2020-06-03 08:33:30 +02:00
Eugen Rochko
5d8398c8b8
Add E2EE API (#13820) 2020-06-02 19:24:53 +02:00
Thibaut Girka
b07c1e615b Fix backup service when user has posted instance-local toots, again 2020-05-21 20:05:46 +02:00
Thibaut Girka
27b5143dc4 Merge branch 'master' into glitch-soc/merge-upstream 2020-05-20 18:22:42 +02:00
ThibG
27ea7c13a5
Fix hashtag search performing account search as well (#13758) 2020-05-14 23:37:37 +02:00
Thibaut Girka
3baacf6993 Merge branch 'master' into glitch-soc/merge-upstream 2020-05-13 22:11:49 +02:00
Takeshi Umeda
d8bad8fbf6
Fix not publishing update of remote timeline (#13745)
* Fix not publishing update of remote timeline

* fix @ missing

* if/unless to if/else
2020-05-12 15:24:35 +02:00
Thibaut Girka
4a70792b4a Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `Gemfile.lock`:
  Not a real conflict, just a glitch-soc-only dependency too close to a
  dependency that got updated upstream. Updated as well.
- `app/models/status.rb`:
  Not a real conflict, just a change too close to glitch-soc-changed code
  for optionally showing boosts in public timelines.
  Applied upstream changes.
- `app/views/layouts/application.html.haml`:
  Upstream a new, static CSS file, conflict due to glitch-soc's theming
  system, include the file regardless of the theme.
- `config/initializers/content_security_policy.rb`:
  Upstream dropped 'unsafe-inline' from the 'style-src' directive, but
  both files are very different. Removed 'unsafe-inline' as well.
2020-05-10 16:19:56 +02:00
Eugen Rochko
4b766f9846
Refactor monkey-patching of Goldfinger (#12561) 2020-05-10 11:41:43 +02:00
Thibaut Girka
a22e6a3683 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/controllers/statuses_controller.rb`:
  Upstream disabled the embed controller for reblogs.
  Not a real conflict, but glitch-soc has an extra line to deal
  with its theming system.
  Ported upstream changes.
- `app/javascript/packs/public.js`:
  Upstream made changes to get rid of most inline CSS, this changes
  javascript for public pages, which in glitch are split between
  different files. Ported those changes.
- `app/models/status.rb`:
  Upstream changed the block check in `Status#permitted_for` to
  include domain-block checks. Not a real conflict with glitch-soc,
  but our scope is slightly different, as our scope for
  unauthenticated access do not include instance-local toots.
  Ported upstream changes.
- `app/serializers/rest/instance_serializer.rb`:
  Not a real conflict, upstream added a new field to the instance
  serializer, the conflict is one line above since we added more of
  that.
  Ported upstream changes.
- `app/views/settings/profiles/show.html.haml`:
  Upstream got rid of most inline CSS and moved hidden elements
  to data attributes in the process, in fields were we have
  different values.
  Ported upstream changes while keeping our glitch-specific
  values.
- `app/views/statuses/_simple_status.html.haml`:
  Upstream got rid of inline CSS on an HAML line we treat
  differently, stripping empty text nodes.
  Ported upstream changes to the style attribute, keeping
  the empty text node stripping behavior.
2020-05-03 21:23:49 +02:00
Eugen Rochko
2744f61696
Fix not being able to resolve public resources in development environment (#13505) 2020-04-25 22:01:08 +02:00
Thibaut Girka
f3eff922a3 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/javascript/mastodon/features/compose/components/poll_form.js`:
  Upstream bumped poll option character limit, but we already had
  a higher one, kept ours.
- `app/validators/poll_validator.rb`:
  Upstream bumped poll option character limit, but we already had
  a higher one, kept ours.
- `config/initializers/content_security_policy.rb`:
  Upstream added a rule, the way we compute ours is different, but
  that added rule has been ported.
- `package.json`:
  No real conflict, dependency update. Performed the same update.
- `yarn.lock`:
  No real conflict, dependency update. Performed the same update.
2020-04-02 20:32:00 +02:00
Eugen Rochko
c3965e28b3
Fix returning results when searching for URL with non-zero offset (#13377)
Fix #13083
2020-04-02 03:39:37 +02:00
Takeshi Umeda
11169367e4
Fix incorrect deletion of local accounts imported by overwriting (#13350) 2020-03-30 20:32:34 +02:00
Thibaut Girka
d98fabf2ee Fix crash when posting with a CW but no text nor media 2020-03-27 22:28:39 +01:00
Thibaut Girka
7b435fd9bf Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/services/post_status_service.rb`:
  CW/sensitive logic fixed upstream, but different in glitch-soc.
  Ported the changes accordingly.
2020-03-27 21:54:44 +01:00
ThibG
f08f880f58
Fix media not being marked sensitive when client sets a CW but no text (#13277)
Mastodon enforces the “sensitive” flag on media attachments whenever a toot
is posted with a Content Warning. However, it does so *after* potentially
converting the Content Warning to toot text (when there is no toot text),
which leads to inconsistent and surprising behavior for API clients.
This commit fixes this inconsistency.
2020-03-25 22:40:58 +01:00
Thibaut Girka
9abb227250 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `README.md`:
  Our README.md files are completely different. Discarded upstream changes.
- `app/javascript/core/admin.js`:
  Updating rails-ujs, no real conflict, but a comment to close to changed
  code. Various glitch-soc-only files have been updated to match those changes,
  though.
- `package.json`:
  No real conflict, just an additional dependency in glitch-soc that was too
  close to something updated upstream. Took upstream's changes.
2020-03-22 16:10:44 +01:00
ThibG
cb12a2cdd3
Fix some timeouts when searching URLs by limiting some database queries (#13253)
Only look up private toots from database if the request failed because of 401,
403 or 404 errors, as those may indicate a private toot, rather than something
that isn't a toot or cannot be processed.
2020-03-12 23:06:43 +01:00
ThibG
b154428e14
Add federation support for the "hide network" preference (#11673)
* Change ActivityPub follower/following collections to not link first page

* Add support for hiding followers and following of remote users

* Switch to using a single `hide_collections` column

* Address code style remarks
2020-03-09 00:10:29 +01:00
Eugen Rochko
9660aa4543
Change local media attachments to perform heavy processing asynchronously (#13210)
Fix #9106
2020-03-08 23:56:18 +01:00
Thibaut Girka
c790ecb14d Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/controllers/api/v1/statuses_controller.rb`:
  Conflict due to upstream adding a new parameter (with_rate_limit),
  too close to glitch-soc's own additional parameter (content_type).
  Added upstream's parameter.
- `app/services/post_status_service.rb`:
  Conflict due to upstream adding a new parameter (rate_limit),
  too close to glitch-soc's own additional parameter (content_type).
  Added upstream's parameter.
- `app/views/settings/preferences/appearance/show.html.haml`:
  Conflict due to us not exposing theme settings here (as we have
  a different flavour/skin menu).
  Took upstream change, while still not exposing theme settings.
- `config/webpack/shared.js`:
  Coding style fixes for a part we have rewritten.
  Discarded upstream changes.
2020-03-08 19:38:53 +01:00
ThibG
5284e29e2f
Fix public posts from silenced accounts not being changed to unlisted visibility (#13096) 2020-03-08 16:11:49 +01:00
ThibG
6185bff4b3
Fix error when searching for URLs that contain the mention syntax (#13151)
Fixes #13150
2020-03-08 15:42:20 +01:00
Eugen Rochko
339ce1c4e9
Add specific rate limits for posting and following (#13172) 2020-03-08 15:17:39 +01:00
Thibaut Girka
85933bc9ff Merge branch 'master' into glitch-soc/merge-upstream 2020-02-25 14:28:13 +01:00
ThibG
cf4fe6caef
Fix misleading error when attempting to re-send a pending follow request (#13133)
Fixes #13131
2020-02-24 21:19:19 +01:00
ThibG
d91946ae13
Fix backups failing when files are missing from media attachments (#13146)
Fixes #13123
2020-02-24 21:18:26 +01:00
ThibG
3704402dcc
Fix duplicate accounts being created when fetching an account for its key only (#13147)
Fixes #13136

When a user's canonical acct domain is different from its id's domain
(WEB_DOMAIN ≠ LOCAL_DOMAIN), two webfinger queries are required to find the
canonical domain from the URI. However, we skip webfinger queries when
updating only the key of a remote user, which led to the creation of a
duplicate account, using the URI's domain instead of the canonical acct: one.
2020-02-24 21:15:53 +01:00
ThibG
983ab40086
Fix previously OStatus-based accounts not being detected as ActivityPub (#13129) 2020-02-22 01:26:41 +01:00
Thibaut Girka
980c004f06 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `Gemfile.lock`:
  No real conflict, glitch-soc-only dependency (redcarpet) too close to an
  upstream one (rdf-normalize)
- `README.md`:
  we have different READMEs, discarded upstream's changes
- `app/views/admin/custom_emojis/index.html.haml`:
  No real conflict, different context because of glitch-soc theming
- `lib/mastodon/statuses_cli.rb`:
  Upstream added code to keep bookmarked statuses, we were already doing so
  with slightly different code. Discarded upstream's changes.
- `package.json`:
  No real conflict, glitch-soc-only dependency (favico.js) too close to
  an upstream one
2020-01-12 15:57:34 +01:00
ThibG
d386d89179 Fix invalid votes from the API being accepted (#12601)
* Fix invalid votes from the API being accepted

Fixes #12556

- Ensure `choice` is an integer instead of silently converting to 0
- Ensure `choice` corresponds to an actual choice of the poll

* Please CodeClimate
2020-01-12 14:17:03 +01:00
Daigo 3 Dango
206dfd7dad Clarify keyword arguments with ** (#12769)
This change is to suppress the warning below on on ruby-2.7.0:

- warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call

https://www.ruby-lang.org/en/news/2019/12/12/separation-of-positional-and-keyword-arguments-in-ruby-3-0/
2020-01-11 05:57:05 +09:00
ThibG
c306978190 Use quality values in Accept field to explicitly prefer JSON over HTML (#12806) 2020-01-10 00:10:29 +01:00
Thibaut Girka
01eaeab56d Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/controllers/application_controller.rb`:
  Conflict due to theming system.
- `app/controllers/oauth/authorizations_controller.rb`:
  Conflict due to theming system.
2020-01-04 23:04:42 +01:00
ThibG
e4d75f238b Fix URL search not returning private toots user has access to (#12742) 2020-01-03 05:01:45 +01:00
Eugen Rochko
f86ee4b59f
Fix IDN mentions not being processed, IDN domains not being rendered (#12715)
This changes the REST API to return unicode domains in the `acct`
attribute instead of punycode, and to render unicode instead of
punycode on public HTML pages as well.

Fix #7812, fix #12246
2019-12-30 19:20:43 +01:00
chr v1.x
7e074610a6 Fix backups stopping due to read timeouts (#12281)
* Make BackupService resilient to read timeouts

If an attachment read times out, assume that the resources is
inaccessible and continue the backup without it. This fixes #12280.

* Both errors on one line
2019-12-30 03:17:54 +01:00
Thibaut Girka
ab5ff638f2 Merge branch 'master' into glitch-soc/master
Conflicts:
- `config/locales/en.yml`
  No real conflict, upstream added a translatable string “too close” to
  one specific to glitch-soc
- `lib/mastodon/statuses_cli.rb`
  Fixes made upstream, while changed in glitch-soc to keep bookmarked statuses
- `package.json`
  No real conflict, additional dependency in glitch-soc
2019-12-19 13:52:54 +01:00
ThibG
b364175e1b Fix link crawler not specifying accepted content-type (#12646)
The link crawler expects HTML documents, so set the `Accept`
header accordingly.

Fixes #12618
2019-12-18 16:56:06 +01:00
Takeshi Umeda
68070e82cc Add signature to exported status (#12649) 2019-12-18 16:55:21 +01:00
Takeshi Umeda
a391eaf4d8 Fix an error when ActivityPub::FetchRemoteStatusService url is called with nil (#12652) 2019-12-18 16:54:03 +01:00
ThibG
2ee5a9d9c3 Clean up OStatus-related codepaths (#12173)
* Remove “protocol” argument and return value, as only ActivityPub is supported

* Remove FetchRemoteAccountService, only use ActivityPub::FetchRemoteAccountService

* Fix tests
2019-12-17 13:32:57 +01:00
Alice Gaudon
668f698077 Admin setting to disable default follows (#12566) 2019-12-16 23:55:50 +01:00
Thibaut Girka
abcba5c198 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- package.json
  Not really a conflict, caused by an additional dependency in glitch-soc.
- yarn.lock
  Not really a conflict, caused by an additional dependency in glitch-soc.
2019-12-07 12:28:59 +01:00
trwnh
1653ae91ce Fix account search with no query (#12549)
* Fix account search with no query

Modeled after #12541. Fix #12548

* fix codeclimate
2019-12-06 19:44:23 +01:00
Eugen Rochko
bbf926b1c0
Fix error when using search API with no query (#12541)
Fix #12462
2019-12-04 04:34:08 +01:00
ThibG
911cc14481 Add follow_request notification type (#12198)
* Add follow_request notification type

The notification type already existed in the backend but was never pushed
to the front-end. This also means translation strings were also available
for the backend, from the notification mailer.

Unlike other notification types, these are off by default, to match what
I remember of Gargron's view on the topic: that follow requests should not
clutter notifications and should instead be reviewed at the user's own
leisure in the dedicated column.

Since follow requests have their own column, I've deemed it unnecessary to
add a specific tab for them in the notification quick filter.

* Show follow request link in single-column if there are pending requests, even if account isn't locked

* Push follow requests from notifications to the follow_requests list

* Offer to accept or reject follow request from the notification

* Redesign follow request notification
2019-12-01 17:25:29 +01:00
Thibaut Girka
2b7158427f Merge branch 'master' into glitch-soc/merge-upstream 2019-11-30 17:29:44 +01:00
ThibG
00c219aa45 Fix empty poll options not being filtered on remote poll update (#12484)
If a poll contains empty options (which is apparently possible on Pleroma),
it is created without them. However, the poll update code doesn't filter
empty options, and thus:
1. Clear known votes, as it assumes the set of options has changed
2. Errors out because it tries adding empty options, which fails validation

This commit fixes that by filtering them out the same way they are filtered
out at poll creation time.
2019-11-26 22:46:31 +01:00
Thibaut Girka
c0f7e21685 Merge branch 'master' into glitch-soc/merge-upstream 2019-11-23 19:03:01 +01:00
Eugen Rochko
a60b602ade
Fix OEmbed discovery not handling different URL variants in query (#12439)
Fix #12433
2019-11-21 19:03:07 +01:00
ThibG
c656cc2191 Fix FetchLinkCardServices crashing on a tags without a target (#12159)
* Add test for links without targets

* Fix FetchLinkCardServices crashing on a tags without a target
2019-11-21 16:04:52 +01:00
ThibG
c4d45765e3 Change domain block behavior to update user records before deleting data (#12247)
When suspending or silencing a domain, one probably wants the silence or
suspend parts to take effect as soon as possible. Deleting media files can
take a while (possibly days), so perform silencing as soon as possible, and
clean up media afterwards.
2019-11-21 11:36:15 +01:00
Thibaut Girka
ff67385cfb Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- README.md
  discarded upstream changes
- app/controllers/api/v1/bookmarks_controller.rb
  finally merged upstream, some code style fixes
  and slightly changed pagination code
- app/controllers/application_controller.rb
  changed upstream to always return HTML error pages
  slight conflict caused by theming code
- app/models/bookmark.rb
  finally merged upstream, no real conflict
- spec/controllers/api/v1/bookmarks_controller_spec.rb
  finally merged upstream, slightly changed pagination code
2019-11-20 15:36:09 +01:00
Eugen Rochko
d14e74eff5
Add cache for OEmbed endpoints to avoid extra HTTP requests (#12403)
* add youtube oembed endpoint

* add check for oembed endpoint

* change unless for a more readable if

* clear blank lines

* endpoint via https

* Fix string literal in condition

* use cache for endpoints

* use cache for endpoints

* clean up and adding check

* clean up and remove redundant return

* add html check

* add false to return

* use double quotes

* use double quotes

* Clean up
2019-11-17 18:40:33 +01:00
ThibG
dfea7368c9 Add bookmarks (#7107)
* Add backend support for bookmarks

Bookmarks behave like favourites, except they aren't shared with other
users and do not have an associated counter.

* Add spec for bookmark endpoints

* Add front-end support for bookmarks

* Introduce OAuth scopes for bookmarks

* Add bookmarks to archive takeout

* Fix migration

* Coding style fixes

* Fix rebase issue

* Update bookmarked_statuses to latest UI changes

* Update bookmark actions to properly reflect status changes in state

* Add bookmarks item to single-column layout

* Make active bookmarks red
2019-11-13 23:02:10 +01:00
ThibG
0d5f3ae331 Fix already-known remote private toots not being searchable by URL (#12336)
* Fix already-known remote private toots not being searchable by URL

Whenever an URL search fails, and an account is provided, try getting a
private status matching that URL from database.

* Fix searching by `id` field instead of `uri`
2019-11-11 00:04:48 +02:00
Thibaut Girka
83774367d2 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/javascript/mastodon/features/compose/components/poll_form.js`
  conflict because of the poll option limit being different than upstream's
2019-11-07 13:43:56 +01:00
ThibG
66c1fe0495 Fix various issues with account migration (#12301)
* Fix being able to follow oneself by moving to an account that was following the old one

* Add specs

* Add spec to catch MoveWorker issue with local followers following both accounts

* Fix move worker breaking when a local account follows both source and target accounts

* Fix migration from remote to local account not sending Undo Follow

* Fix show_reblogs not being preserved for moved account's followers
2019-11-07 09:05:07 +02:00
Thibaut Girka
0ae7d84273 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `package.json`
2019-11-06 14:38:28 +01:00
ThibG
65e13cfacf Add abilityto add oneself to lists (#12271)
* Add ability to add oneself to lists

* Change search results to include oneself when searching through followers

* Mark follow relation as optional in ListAccount
2019-11-04 13:02:01 +01:00
Thibaut Girka
41a98b6543 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- README.md
- app/helpers/statuses_helper.rb
  Upstream moved account helpers to their own file, we had extra
  helpers there, moved too.
- app/lib/sanitize_config.rb
- app/models/user.rb
- app/serializers/initial_state_serializer.rb
- config/locales/simple_form.en.yml
- spec/lib/sanitize_config_spec.rb
2019-10-27 12:14:01 +01:00
nightpool
9762fe382c microformat mentions can have an implicit property (#12189)
See the first example here: http://microformats.org/wiki/microformats2#hyperlinked_person
2019-10-24 22:46:15 +02:00
Thibaut Girka
61631f4751 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- README.md
- app/javascript/styles/mastodon/components.scss
  conflicts caused by image URLs being different
- app/models/status.rb
  as_home_timeline removed, kept glitch-soc-only as_direct_timeline
- app/views/statuses/_simple_status.html.haml
- config/locales/en.yml
  some strings were changed upstream
- spec/models/status_spec.rb
  as_home_timeline removed, kept glitch-soc-only as_direct_timeline
2019-10-10 17:26:08 +02:00
Eugen Rochko
8386d9ec10
Fix hashtag timeline REST API accepting too many hashtags (#12091) 2019-10-07 03:56:21 +02:00
Thibaut Girka
1bbba0016a Merge tag 'v3.0.0' into glitch-soc/merge-upstream 2019-10-04 19:49:43 +02:00
Eugen Rochko
eb83d6256e
Add reason param to POST /api/v1/accounts REST API (#12064)
For approval-required registrations mode
2019-10-03 17:50:59 +02:00
Thibaut Girka
a84aedb7a7 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/models/media_attachment.rb
  Upstream raised max image size from 8MB to 10MB while our limit is
  configurable. Raised the default to 10MB.
2019-10-03 11:10:12 +02:00
ThibG
3a4d994c40 Fix BootstrapTimelineService crashing when bootstrapped accounts are invalid (#12037)
* Add test to handle suspended and missing users in BootstrapTimelineService

* Fix BootstrapTimelineService crashing when bootstrapped accounts are invalid
2019-10-01 15:10:00 +02:00
Eugen Rochko
9ba40a6bfd
Remove HEAD request from fetching link previews (#12028)
It is not really necessary and we need to reduce requests
2019-10-01 04:54:10 +02:00
abcang
9ef5532a7e Fix follow requests N+1 (#12004) 2019-09-30 17:02:03 +02:00
Thibaut Girka
16ff7c5627 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- Gemfile
- Gemfile.lock
- app/controllers/about_controller.rb
- app/controllers/auth/sessions_controller.rb
2019-09-30 12:23:57 +02:00
ThibG
3babf8464b Add voters count support (#11917)
* Add voters count to polls

* Add ActivityPub serialization and parsing of voters count

* Add support for voters count in WebUI

* Move incrementation of voters count out of redis lock

* Reword “voters” to “people”
2019-09-29 22:58:01 +02:00
Eugen Rochko
368a87755b
Fix account migration not affecting followers on origin server (#11980) 2019-09-29 16:23:13 +02:00
Eugen Rochko
3ec80c7aec
Fix preview card image not being re-fetched even if link is re-posted (#11981)
Fix #11956
2019-09-28 01:33:16 +02:00
Eugen Rochko
ab33c4df94
Add exclude_unreviewed param to GET /api/v2/search REST API (#11977)
Make it so normal search returns even unreviewed matches, but
autosuggestions do not.

Fix #11960
2019-09-28 01:02:21 +02:00
ThibG
18b451c0e6 Change silences to always require approval on follow (#11975)
* Change silenced accounts to require approval on follow

* Also require approval for follows by people explicitly muted by target accounts

* Do not auto-accept silenced or muted accounts when switching from locked to unlocked

* Add `follow_requests_count` to verify_credentials

* Show “Follow requests” menu item if needed even if account is locked

* Add tests

* Correctly reflect that follow requests weren't auto-accepted when local account is silenced

* Accept follow requests from user-muted accounts to avoid leaking mutes
2019-09-27 21:13:51 +02:00
Eugen Rochko
2f90a38f44
Fix unreviewed hashtag not being found by exact case-insensitive match (#11976) 2019-09-27 19:40:26 +02:00
Jeong Arm
00d7bdcc2a Fix search error when ElasticSearch is enabled but not available (#11954)
* Fallback to Database search when ES not available

* Prevent double work if ES gives 0 result

* Apply suggestion from code review
2019-09-26 18:06:08 +02:00
Thibaut Girka
5cadb47238 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/controllers/auth/sessions_controller.rb
  Minor conflict due to glitch-soc's theming code
2019-09-18 17:25:56 +02:00
Eugen Rochko
4f6af87906
Change spam check to apply to local accounts and add a threshold (#11806)
Instead of detecting spam on first duplicate message, add a
threshold of 5 such messages to reduce false positives
2019-09-18 12:53:13 +02:00
Eugen Rochko
38dc51b2d6
Fix Move handler queuing jobs that will fail if account is suspended (#11864)
Don't put Move handler on cooldown if it didn't run. Skip unmerging
from timelines to save unnecessary work.
2019-09-17 08:44:45 +02:00
Thibaut Girka
74c5b2bd08 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- Gemfile
- app/controllers/api/v1/search_controller.rb
  Conflict because we changed the number of default results to be
  configurable
- app/lib/settings/scoped_settings.rb
  Addition of a new “noindex” site-wide setting,
  conflict due to our change of the two other site-wide settings
  (default flavour and skin instead of theme)
- spec/controllers/application_controller_spec.rb
  Addition of a new “noindex” site-wide setting,
  conflict due to our change of the two other site-wide settings
  (default flavour and skin instead of theme)
2019-09-13 18:13:43 +02:00
Eugen Rochko
50edf84898
Fix search API not resolving URL when type is given (#11822)
Fix #11771
2019-09-13 16:01:39 +02:00
Eugen Rochko
c5d37f18cb
Change deletes to preserve soft-deleted statuses in unresolved reports (#11805)
Change all account actions except "none" to resolve all unresolved reports

Refactor `SuspendAccountService` to be more readable
2019-09-11 16:32:44 +02:00
Thibaut Girka
5088eb8388 Merge branch 'master' into glitch-soc/merge-upstream 2019-09-05 11:36:41 +02:00
Eugen Rochko
43f56f1291
Change account deletion page to have better explanations (#11753)
Fix deletion of unconfirmed account not freeing up the username

Add prefill of logged-in user's email in the reconfirmation form
2019-09-04 04:13:54 +02:00
Thibaut Girka
3619e904da Fix auto-added media icons when a CW is set but no content text 2019-08-31 18:23:24 +02:00
Thibaut Girka
9044a2b051 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/views/directories/index.html.haml
  Upstream has redesigned the profile directory, and we
  had a glitch-soc-specific change to hide follower counts.
  Ported that change to the new design.
2019-08-30 13:38:41 +02:00
Eugen Rochko
cb447b28c4
Add profile directory to web UI (#11688)
* Add profile directory to web UI

* Add a line of bio to the directory
2019-08-30 00:14:36 +02:00
Thibaut Girka
48b8a1f414 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/models/status.rb
- app/services/remove_status_service.rb
- db/schema.rb

All conflicts were due to the addition of a `deleted_at` attribute
to Statuses and reworked database indexes.
2019-08-29 12:07:50 +02:00
Eugen Rochko
97192d9a77
Fix remote and staff-removed statuses leaving media behind for a day (#11638)
The reason for unattaching media instead of removing it is to support
delete & redraft functionality, but remote or staff-removed statuses
will never be redrafted, so the media should be deleted immediately
2019-08-22 04:17:12 +02:00
Thibaut Girka
1488be7d96 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/controllers/home_controller.rb
- app/controllers/shares_controller.rb
- app/javascript/packs/public.js
- app/models/status.rb
- app/serializers/initial_state_serializer.rb
- app/views/home/index.html.haml
- app/views/layouts/public.html.haml
- app/views/public_timelines/show.html.haml
- app/views/shares/show.html.haml
- app/views/tags/show.html.haml
- config/initializers/content_security_policy.rb
- config/locales/en.yml
- config/webpack/shared.js
- package.json
2019-08-19 21:49:35 +02:00
Eugen Rochko
cc0a55cf9a
Add more accurate hashtag search (#11579)
* Add more accurate hashtag search

Using ElasticSearch to index hashtags with edge n-grams and score
them by usage within the last 7 days since last activity. Only
hashtags that have been reviewed and are listable can appear in
searches, unless they match the query exactly

* Fix search analyzer dropping non-ascii characters
2019-08-18 03:45:51 +02:00
Eugen Rochko
70da6d6630
Fix accounts search by full/partial display name and others (#11580)
- Restrict followers counts to local users to minimize local advantage
- Fix emoji shortcodes causing error in search
- Fix search syntax parse errors not being caught
2019-08-16 13:00:30 +02:00
Eugen Rochko
8fdff2748f
Add more accurate account search (#11537)
* Add more accurate account search

When ElasticSearch is available, a more accurate search is implemented:

- Using edge n-gram index for acct and display name
- Using asciifolding and cjk width normalization on display names
- Using Gaussian decay on account activity for additional scoring (recency)
- Using followers/friends ratio for additional scoring (spamminess)
- Using followers number for additional scoring (size)

The exact match precedence only takes effect when the input conforms
to the username format and the username part of it is complete, i.e.
when the user started typing the domain part.

* Support single-letter usernames

* Fix tests

* Fix not picking up account updates

* Add weights and normalization for scores, skip zero terms queries

* Use local counts for accounts index, adjust search parameters

* Fix mistakes

* Using updated_at of accounts is inadequate for remote accounts
2019-08-16 01:24:03 +02:00
Thibaut Girka
c4ae7aab6f Merge branch 'master' into glitch-soc/merge-upstream 2019-08-12 16:24:22 +02:00
Eugen Rochko
c5661b0f92
Fix account search always returning exact match on paginated results (#11525)
Fix #11365
2019-08-11 13:14:51 +02:00
Thibaut Girka
ebc0521ba8 Merge branch 'master' into glitch-soc/merge-upstream 2019-08-08 08:44:33 +02:00
Eugen Rochko
f51c7c105f
Fix acct URIs with IDN domains not being resolved (#11520)
Fix #11494
2019-08-07 21:14:08 +02:00
ThibG
bced70469a Add domain block notes (#11515)
* Add database columns for adding notes to domain blocks/restrctions

* Add admin UI to set private and public comments when blocking a domain

* Add text for private and public comments on domain blocks

* Show domain block comments in admin UI

* Add comments to the domain block undo page

* Make UnblockDomainService more robust regarding upgraded domain blocks

* Allow editing domain blocks

* Rename button from “undo domain block” to “view domain block” in account admin UI

* Change test to unsilence silenced users from upgraded blocks
2019-08-07 20:20:23 +02:00
Thibaut Girka
04534604c6 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/models/user.rb
- app/serializers/initial_state_serializer.rb
- app/views/admin/dashboard/index.html.haml
- config/locales/simple_form.en.yml
2019-08-07 13:56:48 +02:00
Eugen Rochko
5e35aa8280
Fix non-lowercase hashtags not being picked up by the streaming API (#11508)
Regression from f371b32

Fix hashtag links always being lowercase
2019-08-07 10:01:19 +02:00
Thibaut Girka
24968d20a0 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/controllers/directories_controller.rb
- package.json
- yarn.lock
2019-07-30 12:22:33 +02:00
Eugen Rochko
24552b5160
Add whitelist mode (#11291) 2019-07-30 11:10:46 +02:00
Thibaut Girka
bca3825c17 Merge branch 'master' into glitch-soc/merge-upstream 2019-07-28 16:28:05 +02:00
Eugen Rochko
f371b32137
Change hashtags to preserve first-used casing (#11416) 2019-07-28 05:59:51 +02:00
Eugen Rochko
b9fbcbfe4e
Add search syntax for operators and phrases (#11411) 2019-07-27 04:42:08 +02:00
ThibG
92569ffde8 Fix invites not being disabled upon account suspension (#11412)
* Disable invite links from disabled/suspended users

* Add has_many invites relationship to users

* Destroy unused invites when suspending an account
2019-07-26 18:55:33 +02:00
Eugen Rochko
4eeff26533
Change account domain block to clear out notifications and follows (#11393) 2019-07-25 04:17:35 +02:00
Thibaut Girka
444796b69b Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/controllers/application_controller.rb
- app/controllers/auth/confirmations_controller.rb
- app/controllers/auth/sessions_controller.rb
- app/controllers/settings/deletes_controller.rb
- app/controllers/settings/two_factor_authentication/recovery_codes_controller.rb
2019-07-23 10:51:07 +02:00
Eugen Rochko
bd87e66679
Remove WebSub subscriptions (#11303) 2019-07-21 04:08:00 +02:00
Thibaut Girka
249991c498 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- Gemfile.lock
- app/controllers/accounts_controller.rb
- app/controllers/admin/dashboard_controller.rb
- app/controllers/follower_accounts_controller.rb
- app/controllers/following_accounts_controller.rb
- app/controllers/remote_follow_controller.rb
- app/controllers/stream_entries_controller.rb
- app/controllers/tags_controller.rb
- app/javascript/packs/public.js
- app/lib/sanitize_config.rb
- app/models/account.rb
- app/models/form/admin_settings.rb
- app/models/media_attachment.rb
- app/models/stream_entry.rb
- app/models/user.rb
- app/serializers/initial_state_serializer.rb
- app/services/batched_remove_status_service.rb
- app/services/post_status_service.rb
- app/services/process_mentions_service.rb
- app/services/reblog_service.rb
- app/services/remove_status_service.rb
- app/views/admin/settings/edit.html.haml
- config/locales/simple_form.pl.yml
- config/settings.yml
- docker-compose.yml
2019-07-19 18:26:49 +02:00
ThibG
2f813b7ea1 Disable LDSigning when AUTHORIZED_FETCH is set to true (#11295) 2019-07-15 02:29:39 +02:00
ThibG
3595ce6325 Fix leaking private statuses the admin account follows (#11300)
Now that the request is signed, it can return private toots. Do not leak them.
2019-07-15 02:29:04 +02:00
ThibG
99c1f0811b Fix BlockService trying to reject incorrect follow request (#11288)
Fixes #11148
2019-07-15 00:48:50 +02:00
ThibG
bd77fd6ff3 Fix BackupService crashing when an attachment is missing (#11241)
* Fix BackupService crashing when an attachment is missing

For various reasons such as admin error or out-of-sync media and
database backups, it might be possible for local attachments to be lost.

This commit allows the BackupService to continue its work even if some media
file is missing.

* Change error message
2019-07-15 00:48:49 +02:00
Eugen Rochko
6ff67be0f6
Add a spam check (#11217)
* Add a spam check

* Use Nilsimsa to generate locality-sensitive hashes and compare using Levenshtein distance

* Add more tests

* Add exemption when the message is a reply to something that mentions the sender

* Use Nilsimsa Compare Value instead of Levenshtein distance

* Use MD5 for messages shorter than 10 characters

* Add message to automated report, do not add non-public statuses to
automated report, add trust level to accounts and make unsilencing
raise the trust level to prevent repeated spam checks on that account

* Expire spam check data after 3 months

* Add support for local statuses, reduce expiration to 1 week, always create a report

* Add content warnings to the spam check and exempt empty statuses

* Change Nilsimsa threshold to 95 and make sure removed statuses are removed from the spam check

* Add all matched statuses into automatic report
2019-07-13 16:45:50 +02:00
ThibG
4e1260feaa Fix BlockService trying to reject incorrect follow request (#11288)
Fixes #11148
2019-07-11 14:50:27 +02:00
Eugen Rochko
4e8dcc5dbb
Add HTTP signatures to all outgoing ActivityPub GET requests (#11284) 2019-07-11 14:49:55 +02:00
Eugen Rochko
5d3feed191
Refactor fetching of remote resources (#11251) 2019-07-10 18:59:28 +02:00
ThibG
85eb418e1f Fix handling of webfinger redirects in ResolveAccountService (#11279) 2019-07-10 17:10:12 +02:00
Eugen Rochko
4e92183227
Refactor domain block checks (#11268) 2019-07-09 03:27:35 +02:00
Eugen Rochko
63c7fe8e48
Refactor controllers for statuses, accounts, and more (#11249) 2019-07-08 12:03:45 +02:00
ThibG
87c1ad4ea3 Fix BackupService crashing when an attachment is missing (#11241)
* Fix BackupService crashing when an attachment is missing

For various reasons such as admin error or out-of-sync media and
database backups, it might be possible for local attachments to be lost.

This commit allows the BackupService to continue its work even if some media
file is missing.

* Change error message
2019-07-08 02:24:25 +02:00
ThibG
fce885f271
Merge pull request #1152 from ThibG/glitch-soc/merge-upstream
Merge upstream changes
2019-07-07 22:26:10 +02:00
Eugen Rochko
b851456139
Remove Atom feeds and old URLs in the form of GET /:username/updates/:id (#11247) 2019-07-07 16:16:51 +02:00
Eugen Rochko
23aeef52cc
Remove Salmon and PubSubHubbub (#11205)
* Remove Salmon and PubSubHubbub endpoints

* Add error when trying to follow OStatus accounts

* Fix new accounts not being created in ResolveAccountService
2019-07-06 23:26:16 +02:00
Thibaut Girka
e5f4d83e60 Allow serializing local-only toots in backup service
Fixes #1153
2019-07-05 21:50:19 +02:00
Thibaut Girka
6ab84c12a7 Merge branch 'master' into glitch-soc/merge-upstream 2019-07-04 16:21:39 +02:00
Eugen Rochko
2cfa427ea7
Change domain block behaviour to prevent creation of accounts from suspended domains (#11219) 2019-07-01 19:13:30 +02:00
Thibaut Girka
ca17bae904 Use a redis-cached feed for the DM timeline 2019-06-27 16:44:12 +02:00
Thibaut Girka
aaec64a500 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/controllers/settings/preferences_controller.rb
- app/lib/user_settings_decorator.rb
- app/models/user.rb
- config/locales/simple_form.en.yml
2019-06-26 23:19:22 +02:00