Merge pull request from GHSA-4gj5-xj97-j8fp
* ci: add build check to pull requests * Build(deps-dev): Bump @types/node from 20.10.6 to 20.11.24 (#91) Bumps [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) from 20.10.6 to 20.11.24. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node) --- updated-dependencies: - dependency-name: "@types/node" dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat: image compression (#101) * feat: image compression * fix: not compress svg Maybe we should add a function to disable this fix. Since I noticed that if you compress svg in webp it becomes magically smaller. * Configuration page (#104) * feat: image compression * feat: configuration page * refactor: json stringify and change engine buttons to ordered list * fix: engine distributor matching * fix: formatting * build: update txtdot version to 1.7.0 * fix: configuration page title * doc: features (#102) * fix: ssrf GHSA-4gj5-xj97-j8fp Doesn't fully correct the error! You need to configure the server to block requests. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This commit is contained in:
Artemy Egorov
GitHub
parent
f241a46e05
commit
7c72d985f7
22
.env.example
22
.env.example
@ -1,14 +1,20 @@
|
||||
HOST=127.0.0.1 # 0.0.0.0 if txtdot is not behind reverse proxy
|
||||
# Server settings
|
||||
|
||||
HOST=0.0.0.0
|
||||
PORT=8080
|
||||
TIMEOUT=0 # 0 means no timeout
|
||||
REVERSE_PROXY=false # only with reverse proxy; see docs
|
||||
|
||||
TIMEOUT=0 # Connection timout 0 (no timout)
|
||||
|
||||
REVERSE_PROXY=true # only for reverse proxy; see docs
|
||||
# Features
|
||||
|
||||
## Proxy
|
||||
PROXY_RES=true
|
||||
SWAGGER=false # whether to add API docs route or not
|
||||
|
||||
# Search
|
||||
IMG_COMPRESS=true # enable image compressing; proxy_res is required
|
||||
|
||||
SEARCH_ENABLED=false # If enabled then you need >
|
||||
SEARX_URL="" # Base url of searxng instance without /search, etc
|
||||
## Documentation
|
||||
SWAGGER=false # whether to add API docs route
|
||||
|
||||
## Search
|
||||
SEARCH_ENABLED=false # searx_url is required when enabled
|
||||
SEARX_URL="" # SearXNG base URL, e.g. https://searx.dc09.ru
|
||||
19
.github/workflows/build.yml
vendored
Normal file
19
.github/workflows/build.yml
vendored
Normal file
@ -0,0 +1,19 @@
|
||||
name: build-check
|
||||
on: pull_request
|
||||
jobs:
|
||||
build-check:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v2
|
||||
|
||||
- name: Set up Node.js
|
||||
uses: actions/setup-node@v2
|
||||
with:
|
||||
node-version: '14'
|
||||
|
||||
- name: Install dependencies
|
||||
run: npm install
|
||||
|
||||
- name: Start build
|
||||
run: npm run build
|
||||
2
.github/workflows/format-check.yml
vendored
2
.github/workflows/format-check.yml
vendored
@ -1,7 +1,7 @@
|
||||
name: format-check
|
||||
on: pull_request
|
||||
jobs:
|
||||
check-formatting:
|
||||
format-check:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout code
|
||||
|
||||
83
README.md
83
README.md
@ -11,28 +11,32 @@
|
||||
|
||||
HTTP proxy that parses only text, links and pictures from pages
|
||||
reducing internet traffic, removing ads and heavy scripts.
|
||||
Mozilla's Readability library is used under the hood.
|
||||
|
||||
Uses [Mozilla's readability.js](https://github.com/mozilla/readability),
|
||||
[🔗 linkedom](https://github.com/WebReflection/linkedom),
|
||||
[Fastify web framework](https://github.com/fastify/fastify).
|
||||
## Features
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
npm install
|
||||
```
|
||||
- Server-side page simplification
|
||||
- Media proxy
|
||||
- Image compression with Sharp
|
||||
- Search with SearXNG
|
||||
- Custom parsers for StackOverflow and SearXNG
|
||||
- Handy API endpoints
|
||||
- No client JavaScript
|
||||
- Some kind of Material Design 3
|
||||
|
||||
## Running
|
||||
|
||||
### Dev
|
||||
### Development
|
||||
|
||||
```bash
|
||||
npm install
|
||||
npm run dev
|
||||
```
|
||||
|
||||
### Prod
|
||||
### Production
|
||||
|
||||
```bash
|
||||
npm install
|
||||
npm run build
|
||||
npm run start
|
||||
```
|
||||
@ -40,6 +44,63 @@ npm run start
|
||||
### Docker
|
||||
|
||||
```bash
|
||||
docker compose build
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
## Screenshots
|
||||
|
||||
<div align="center">
|
||||
<img src="https://raw.githubusercontent.com/TxtDot/.github/main/imgs/ui_url_input.png" alt="Main page with URL input field">
|
||||
<img src="https://raw.githubusercontent.com/TxtDot/.github/main/imgs/ui_search_page.png" alt="SearXNG search results page">
|
||||
</div>
|
||||
|
||||
## Performance tests
|
||||
|
||||
txtdot is a great tool in case of slow internet connection or weak signal.
|
||||
Here is the comparision of performance metrics from pagespeed.web.dev
|
||||
between original page and proxied one.
|
||||
|
||||
"Mobile" test includes "Slow 4G" artificial network throttling.
|
||||
|
||||
<details>
|
||||
<summary>Expand</summary>
|
||||
|
||||
| | Original page | Proxied through txtdot |
|
||||
| :------------------------------- | :-------------------: | :--------------------: |
|
||||
| [Habr][habr-link] Desktop | ![56%][habr-do-img] | ![99%][habr-dt-img] |
|
||||
| [Habr][habr-link] Mobile | ![21%][habr-mo-img] | ![100%][habr-mt-img] |
|
||||
| [Medium][medium-link] Desktop | ![44%][medium-do-img] | ![100%][medium-dt-img] |
|
||||
| [Medium][medium-link] Mobile | ![36%][medium-mo-img] | ![100%][medium-mt-img] |
|
||||
| [Nginx Blog][nginx-link] Desktop | ![53%][nginx-do-img] | ![100%][nginx-dt-img] |
|
||||
| [Nginx Blog][nginx-link] Mobile | ![26%][nginx-mo-img] | ![100%][nginx-mt-img] |
|
||||
|
||||
[habr-link]: https://habr.com/ru/articles/780692/
|
||||
[habr-do-img]: https://raw.githubusercontent.com/TxtDot/.github/main/tests/habr/desktop_orig.png
|
||||
[habr-dt-img]: https://raw.githubusercontent.com/TxtDot/.github/main/tests/habr/desktop_txtdot.png
|
||||
[habr-mo-img]: https://raw.githubusercontent.com/TxtDot/.github/main/tests/habr/mobile_orig.png
|
||||
[habr-mt-img]: https://raw.githubusercontent.com/TxtDot/.github/main/tests/habr/mobile_txtdot.png
|
||||
[medium-link]: https://levelup.gitconnected.com/proxy-servers-how-proxies-work-0ec083fc1030
|
||||
[medium-do-img]: https://raw.githubusercontent.com/TxtDot/.github/main/tests/medium/desktop_orig.png
|
||||
[medium-dt-img]: https://raw.githubusercontent.com/TxtDot/.github/main/tests/medium/desktop_txtdot.png
|
||||
[medium-mo-img]: https://raw.githubusercontent.com/TxtDot/.github/main/tests/medium/mobile_orig.png
|
||||
[medium-mt-img]: https://raw.githubusercontent.com/TxtDot/.github/main/tests/medium/mobile_txtdot.png
|
||||
[nginx-link]: https://www.nginx.com/blog/rate-limiting-nginx/
|
||||
[nginx-do-img]: https://raw.githubusercontent.com/TxtDot/.github/main/tests/nginx-blog/desktop_orig.png
|
||||
[nginx-dt-img]: https://raw.githubusercontent.com/TxtDot/.github/main/tests/nginx-blog/desktop_txtdot.png
|
||||
[nginx-mo-img]: https://raw.githubusercontent.com/TxtDot/.github/main/tests/nginx-blog/mobile_orig.png
|
||||
[nginx-mt-img]: https://raw.githubusercontent.com/TxtDot/.github/main/tests/nginx-blog/mobile_txtdot.png
|
||||
|
||||
</details>
|
||||
|
||||
## Credits
|
||||
|
||||
- [Readability.js](https://github.com/mozilla/readability)
|
||||
- [🔗 LinkeDOM](https://github.com/WebReflection/linkedom)
|
||||
- [Fastify web framework](https://github.com/fastify/fastify)
|
||||
- [EJS](https://github.com/mde/ejs)
|
||||
- [Axios](https://github.com/axios/axios)
|
||||
- [DOMPurify](https://github.com/cure53/DOMPurify)
|
||||
- [Sharp](https://github.com/lovell/sharp)
|
||||
- [MicroMatch](https://github.com/micromatch/micromatch)
|
||||
- [RouteParser](https://github.com/rcs/route-parser)
|
||||
- [IconvLite](https://github.com/ashtuchkin/iconv-lite)
|
||||
|
||||
542
package-lock.json
generated
542
package-lock.json
generated
@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "txtdot",
|
||||
"version": "1.6.1",
|
||||
"version": "1.7.0",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "txtdot",
|
||||
"version": "1.6.1",
|
||||
"version": "1.7.0",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@fastify/static": "^6.12.0",
|
||||
@ -24,14 +24,15 @@
|
||||
"json-schema-to-ts": "^3.0.0",
|
||||
"linkedom": "^0.16.8",
|
||||
"micromatch": "^4.0.5",
|
||||
"route-parser": "^0.0.5"
|
||||
"route-parser": "^0.0.5",
|
||||
"sharp": "^0.33.2"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/dompurify": "^3.0.5",
|
||||
"@types/ejs": "^3.1.5",
|
||||
"@types/jsdom": "^21.1.6",
|
||||
"@types/micromatch": "^4.0.6",
|
||||
"@types/node": "^20.11.20",
|
||||
"@types/node": "^20.11.24",
|
||||
"@types/route-parser": "^0.1.7",
|
||||
"@typescript-eslint/eslint-plugin": "^7.1.0",
|
||||
"@typescript-eslint/parser": "^7.1.0",
|
||||
@ -63,6 +64,15 @@
|
||||
"node": ">=6.9.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@emnapi/runtime": {
|
||||
"version": "0.45.0",
|
||||
"resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-0.45.0.tgz",
|
||||
"integrity": "sha512-Txumi3td7J4A/xTTwlssKieHKTGl3j4A1tglBx72auZ49YK7ePY6XZricgIg9mnZT4xPfA+UPCUdnhRuEFDL+w==",
|
||||
"optional": true,
|
||||
"dependencies": {
|
||||
"tslib": "^2.4.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@eslint-community/eslint-utils": {
|
||||
"version": "4.4.0",
|
||||
"resolved": "https://registry.npmjs.org/@eslint-community/eslint-utils/-/eslint-utils-4.4.0.tgz",
|
||||
@ -313,6 +323,437 @@
|
||||
"integrity": "sha512-6EwiSjwWYP7pTckG6I5eyFANjPhmPjUX9JRLUSfNPC7FX7zK9gyZAfUEaECL6ALTpGX5AjnBq3C9XmVWPitNpw==",
|
||||
"dev": true
|
||||
},
|
||||
"node_modules/@img/sharp-darwin-arm64": {
|
||||
"version": "0.33.2",
|
||||
"resolved": "https://registry.npmjs.org/@img/sharp-darwin-arm64/-/sharp-darwin-arm64-0.33.2.tgz",
|
||||
"integrity": "sha512-itHBs1rPmsmGF9p4qRe++CzCgd+kFYktnsoR1sbIAfsRMrJZau0Tt1AH9KVnufc2/tU02Gf6Ibujx+15qRE03w==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"optional": true,
|
||||
"os": [
|
||||
"darwin"
|
||||
],
|
||||
"engines": {
|
||||
"glibc": ">=2.26",
|
||||
"node": "^18.17.0 || ^20.3.0 || >=21.0.0",
|
||||
"npm": ">=9.6.5",
|
||||
"pnpm": ">=7.1.0",
|
||||
"yarn": ">=3.2.0"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://opencollective.com/libvips"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"@img/sharp-libvips-darwin-arm64": "1.0.1"
|
||||
}
|
||||
},
|
||||
"node_modules/@img/sharp-darwin-x64": {
|
||||
"version": "0.33.2",
|
||||
"resolved": "https://registry.npmjs.org/@img/sharp-darwin-x64/-/sharp-darwin-x64-0.33.2.tgz",
|
||||
"integrity": "sha512-/rK/69Rrp9x5kaWBjVN07KixZanRr+W1OiyKdXcbjQD6KbW+obaTeBBtLUAtbBsnlTTmWthw99xqoOS7SsySDg==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"optional": true,
|
||||
"os": [
|
||||
"darwin"
|
||||
],
|
||||
"engines": {
|
||||
"glibc": ">=2.26",
|
||||
"node": "^18.17.0 || ^20.3.0 || >=21.0.0",
|
||||
"npm": ">=9.6.5",
|
||||
"pnpm": ">=7.1.0",
|
||||
"yarn": ">=3.2.0"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://opencollective.com/libvips"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"@img/sharp-libvips-darwin-x64": "1.0.1"
|
||||
}
|
||||
},
|
||||
"node_modules/@img/sharp-libvips-darwin-arm64": {
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/@img/sharp-libvips-darwin-arm64/-/sharp-libvips-darwin-arm64-1.0.1.tgz",
|
||||
"integrity": "sha512-kQyrSNd6lmBV7O0BUiyu/OEw9yeNGFbQhbxswS1i6rMDwBBSX+e+rPzu3S+MwAiGU3HdLze3PanQ4Xkfemgzcw==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"optional": true,
|
||||
"os": [
|
||||
"darwin"
|
||||
],
|
||||
"engines": {
|
||||
"macos": ">=11",
|
||||
"npm": ">=9.6.5",
|
||||
"pnpm": ">=7.1.0",
|
||||
"yarn": ">=3.2.0"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://opencollective.com/libvips"
|
||||
}
|
||||
},
|
||||
"node_modules/@img/sharp-libvips-darwin-x64": {
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/@img/sharp-libvips-darwin-x64/-/sharp-libvips-darwin-x64-1.0.1.tgz",
|
||||
"integrity": "sha512-eVU/JYLPVjhhrd8Tk6gosl5pVlvsqiFlt50wotCvdkFGf+mDNBJxMh+bvav+Wt3EBnNZWq8Sp2I7XfSjm8siog==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"optional": true,
|
||||
"os": [
|
||||
"darwin"
|
||||
],
|
||||
"engines": {
|
||||
"macos": ">=10.13",
|
||||
"npm": ">=9.6.5",
|
||||
"pnpm": ">=7.1.0",
|
||||
"yarn": ">=3.2.0"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://opencollective.com/libvips"
|
||||
}
|
||||
},
|
||||
"node_modules/@img/sharp-libvips-linux-arm": {
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-arm/-/sharp-libvips-linux-arm-1.0.1.tgz",
|
||||
"integrity": "sha512-FtdMvR4R99FTsD53IA3LxYGghQ82t3yt0ZQ93WMZ2xV3dqrb0E8zq4VHaTOuLEAuA83oDawHV3fd+BsAPadHIQ==",
|
||||
"cpu": [
|
||||
"arm"
|
||||
],
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"glibc": ">=2.28",
|
||||
"npm": ">=9.6.5",
|
||||
"pnpm": ">=7.1.0",
|
||||
"yarn": ">=3.2.0"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://opencollective.com/libvips"
|
||||
}
|
||||
},
|
||||
"node_modules/@img/sharp-libvips-linux-arm64": {
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-arm64/-/sharp-libvips-linux-arm64-1.0.1.tgz",
|
||||
"integrity": "sha512-bnGG+MJjdX70mAQcSLxgeJco11G+MxTz+ebxlz8Y3dxyeb3Nkl7LgLI0mXupoO+u1wRNx/iRj5yHtzA4sde1yA==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"glibc": ">=2.26",
|
||||
"npm": ">=9.6.5",
|
||||
"pnpm": ">=7.1.0",
|
||||
"yarn": ">=3.2.0"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://opencollective.com/libvips"
|
||||
}
|
||||
},
|
||||
"node_modules/@img/sharp-libvips-linux-s390x": {
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-s390x/-/sharp-libvips-linux-s390x-1.0.1.tgz",
|
||||
"integrity": "sha512-3+rzfAR1YpMOeA2zZNp+aYEzGNWK4zF3+sdMxuCS3ey9HhDbJ66w6hDSHDMoap32DueFwhhs3vwooAB2MaK4XQ==",
|
||||
"cpu": [
|
||||
"s390x"
|
||||
],
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"glibc": ">=2.28",
|
||||
"npm": ">=9.6.5",
|
||||
"pnpm": ">=7.1.0",
|
||||
"yarn": ">=3.2.0"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://opencollective.com/libvips"
|
||||
}
|
||||
},
|
||||
"node_modules/@img/sharp-libvips-linux-x64": {
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-x64/-/sharp-libvips-linux-x64-1.0.1.tgz",
|
||||
"integrity": "sha512-3NR1mxFsaSgMMzz1bAnnKbSAI+lHXVTqAHgc1bgzjHuXjo4hlscpUxc0vFSAPKI3yuzdzcZOkq7nDPrP2F8Jgw==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"glibc": ">=2.26",
|
||||
"npm": ">=9.6.5",
|
||||
"pnpm": ">=7.1.0",
|
||||
"yarn": ">=3.2.0"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://opencollective.com/libvips"
|
||||
}
|
||||
},
|
||||
"node_modules/@img/sharp-libvips-linuxmusl-arm64": {
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/@img/sharp-libvips-linuxmusl-arm64/-/sharp-libvips-linuxmusl-arm64-1.0.1.tgz",
|
||||
"integrity": "sha512-5aBRcjHDG/T6jwC3Edl3lP8nl9U2Yo8+oTl5drd1dh9Z1EBfzUKAJFUDTDisDjUwc7N4AjnPGfCA3jl3hY8uDg==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"musl": ">=1.2.2",
|
||||
"npm": ">=9.6.5",
|
||||
"pnpm": ">=7.1.0",
|
||||
"yarn": ">=3.2.0"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://opencollective.com/libvips"
|
||||
}
|
||||
},
|
||||
"node_modules/@img/sharp-libvips-linuxmusl-x64": {
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/@img/sharp-libvips-linuxmusl-x64/-/sharp-libvips-linuxmusl-x64-1.0.1.tgz",
|
||||
"integrity": "sha512-dcT7inI9DBFK6ovfeWRe3hG30h51cBAP5JXlZfx6pzc/Mnf9HFCQDLtYf4MCBjxaaTfjCCjkBxcy3XzOAo5txw==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"musl": ">=1.2.2",
|
||||
"npm": ">=9.6.5",
|
||||
"pnpm": ">=7.1.0",
|
||||
"yarn": ">=3.2.0"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://opencollective.com/libvips"
|
||||
}
|
||||
},
|
||||
"node_modules/@img/sharp-linux-arm": {
|
||||
"version": "0.33.2",
|
||||
"resolved": "https://registry.npmjs.org/@img/sharp-linux-arm/-/sharp-linux-arm-0.33.2.tgz",
|
||||
"integrity": "sha512-Fndk/4Zq3vAc4G/qyfXASbS3HBZbKrlnKZLEJzPLrXoJuipFNNwTes71+Ki1hwYW5lch26niRYoZFAtZVf3EGA==",
|
||||
"cpu": [
|
||||
"arm"
|
||||
],
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"glibc": ">=2.28",
|
||||
"node": "^18.17.0 || ^20.3.0 || >=21.0.0",
|
||||
"npm": ">=9.6.5",
|
||||
"pnpm": ">=7.1.0",
|
||||
"yarn": ">=3.2.0"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://opencollective.com/libvips"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"@img/sharp-libvips-linux-arm": "1.0.1"
|
||||
}
|
||||
},
|
||||
"node_modules/@img/sharp-linux-arm64": {
|
||||
"version": "0.33.2",
|
||||
"resolved": "https://registry.npmjs.org/@img/sharp-linux-arm64/-/sharp-linux-arm64-0.33.2.tgz",
|
||||
"integrity": "sha512-pz0NNo882vVfqJ0yNInuG9YH71smP4gRSdeL09ukC2YLE6ZyZePAlWKEHgAzJGTiOh8Qkaov6mMIMlEhmLdKew==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"glibc": ">=2.26",
|
||||
"node": "^18.17.0 || ^20.3.0 || >=21.0.0",
|
||||
"npm": ">=9.6.5",
|
||||
"pnpm": ">=7.1.0",
|
||||
"yarn": ">=3.2.0"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://opencollective.com/libvips"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"@img/sharp-libvips-linux-arm64": "1.0.1"
|
||||
}
|
||||
},
|
||||
"node_modules/@img/sharp-linux-s390x": {
|
||||
"version": "0.33.2",
|
||||
"resolved": "https://registry.npmjs.org/@img/sharp-linux-s390x/-/sharp-linux-s390x-0.33.2.tgz",
|
||||
"integrity": "sha512-MBoInDXDppMfhSzbMmOQtGfloVAflS2rP1qPcUIiITMi36Mm5YR7r0ASND99razjQUpHTzjrU1flO76hKvP5RA==",
|
||||
"cpu": [
|
||||
"s390x"
|
||||
],
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"glibc": ">=2.28",
|
||||
"node": "^18.17.0 || ^20.3.0 || >=21.0.0",
|
||||
"npm": ">=9.6.5",
|
||||
"pnpm": ">=7.1.0",
|
||||
"yarn": ">=3.2.0"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://opencollective.com/libvips"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"@img/sharp-libvips-linux-s390x": "1.0.1"
|
||||
}
|
||||
},
|
||||
"node_modules/@img/sharp-linux-x64": {
|
||||
"version": "0.33.2",
|
||||
"resolved": "https://registry.npmjs.org/@img/sharp-linux-x64/-/sharp-linux-x64-0.33.2.tgz",
|
||||
"integrity": "sha512-xUT82H5IbXewKkeF5aiooajoO1tQV4PnKfS/OZtb5DDdxS/FCI/uXTVZ35GQ97RZXsycojz/AJ0asoz6p2/H/A==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"glibc": ">=2.26",
|
||||
"node": "^18.17.0 || ^20.3.0 || >=21.0.0",
|
||||
"npm": ">=9.6.5",
|
||||
"pnpm": ">=7.1.0",
|
||||
"yarn": ">=3.2.0"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://opencollective.com/libvips"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"@img/sharp-libvips-linux-x64": "1.0.1"
|
||||
}
|
||||
},
|
||||
"node_modules/@img/sharp-linuxmusl-arm64": {
|
||||
"version": "0.33.2",
|
||||
"resolved": "https://registry.npmjs.org/@img/sharp-linuxmusl-arm64/-/sharp-linuxmusl-arm64-0.33.2.tgz",
|
||||
"integrity": "sha512-F+0z8JCu/UnMzg8IYW1TMeiViIWBVg7IWP6nE0p5S5EPQxlLd76c8jYemG21X99UzFwgkRo5yz2DS+zbrnxZeA==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"musl": ">=1.2.2",
|
||||
"node": "^18.17.0 || ^20.3.0 || >=21.0.0",
|
||||
"npm": ">=9.6.5",
|
||||
"pnpm": ">=7.1.0",
|
||||
"yarn": ">=3.2.0"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://opencollective.com/libvips"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"@img/sharp-libvips-linuxmusl-arm64": "1.0.1"
|
||||
}
|
||||
},
|
||||
"node_modules/@img/sharp-linuxmusl-x64": {
|
||||
"version": "0.33.2",
|
||||
"resolved": "https://registry.npmjs.org/@img/sharp-linuxmusl-x64/-/sharp-linuxmusl-x64-0.33.2.tgz",
|
||||
"integrity": "sha512-+ZLE3SQmSL+Fn1gmSaM8uFusW5Y3J9VOf+wMGNnTtJUMUxFhv+P4UPaYEYT8tqnyYVaOVGgMN/zsOxn9pSsO2A==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"musl": ">=1.2.2",
|
||||
"node": "^18.17.0 || ^20.3.0 || >=21.0.0",
|
||||
"npm": ">=9.6.5",
|
||||
"pnpm": ">=7.1.0",
|
||||
"yarn": ">=3.2.0"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://opencollective.com/libvips"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"@img/sharp-libvips-linuxmusl-x64": "1.0.1"
|
||||
}
|
||||
},
|
||||
"node_modules/@img/sharp-wasm32": {
|
||||
"version": "0.33.2",
|
||||
"resolved": "https://registry.npmjs.org/@img/sharp-wasm32/-/sharp-wasm32-0.33.2.tgz",
|
||||
"integrity": "sha512-fLbTaESVKuQcpm8ffgBD7jLb/CQLcATju/jxtTXR1XCLwbOQt+OL5zPHSDMmp2JZIeq82e18yE0Vv7zh6+6BfQ==",
|
||||
"cpu": [
|
||||
"wasm32"
|
||||
],
|
||||
"optional": true,
|
||||
"dependencies": {
|
||||
"@emnapi/runtime": "^0.45.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": "^18.17.0 || ^20.3.0 || >=21.0.0",
|
||||
"npm": ">=9.6.5",
|
||||
"pnpm": ">=7.1.0",
|
||||
"yarn": ">=3.2.0"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://opencollective.com/libvips"
|
||||
}
|
||||
},
|
||||
"node_modules/@img/sharp-win32-ia32": {
|
||||
"version": "0.33.2",
|
||||
"resolved": "https://registry.npmjs.org/@img/sharp-win32-ia32/-/sharp-win32-ia32-0.33.2.tgz",
|
||||
"integrity": "sha512-okBpql96hIGuZ4lN3+nsAjGeggxKm7hIRu9zyec0lnfB8E7Z6p95BuRZzDDXZOl2e8UmR4RhYt631i7mfmKU8g==",
|
||||
"cpu": [
|
||||
"ia32"
|
||||
],
|
||||
"optional": true,
|
||||
"os": [
|
||||
"win32"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^18.17.0 || ^20.3.0 || >=21.0.0",
|
||||
"npm": ">=9.6.5",
|
||||
"pnpm": ">=7.1.0",
|
||||
"yarn": ">=3.2.0"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://opencollective.com/libvips"
|
||||
}
|
||||
},
|
||||
"node_modules/@img/sharp-win32-x64": {
|
||||
"version": "0.33.2",
|
||||
"resolved": "https://registry.npmjs.org/@img/sharp-win32-x64/-/sharp-win32-x64-0.33.2.tgz",
|
||||
"integrity": "sha512-E4magOks77DK47FwHUIGH0RYWSgRBfGdK56kIHSVeB9uIS4pPFr4N2kIVsXdQQo4LzOsENKV5KAhRlRL7eMAdg==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"optional": true,
|
||||
"os": [
|
||||
"win32"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^18.17.0 || ^20.3.0 || >=21.0.0",
|
||||
"npm": ">=9.6.5",
|
||||
"pnpm": ">=7.1.0",
|
||||
"yarn": ">=3.2.0"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://opencollective.com/libvips"
|
||||
}
|
||||
},
|
||||
"node_modules/@lukeed/ms": {
|
||||
"version": "2.0.2",
|
||||
"resolved": "https://registry.npmjs.org/@lukeed/ms/-/ms-2.0.2.tgz",
|
||||
@ -411,9 +852,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@types/node": {
|
||||
"version": "20.11.22",
|
||||
"resolved": "https://registry.npmjs.org/@types/node/-/node-20.11.22.tgz",
|
||||
"integrity": "sha512-/G+IxWxma6V3E+pqK1tSl2Fo1kl41pK1yeCyDsgkF9WlVAme4j5ISYM2zR11bgLFJGLN5sVK40T4RJNuiZbEjA==",
|
||||
"version": "20.11.24",
|
||||
"resolved": "https://registry.npmjs.org/@types/node/-/node-20.11.24.tgz",
|
||||
"integrity": "sha512-Kza43ewS3xoLgCEpQrsT+xRo/EJej1y0kVYGiLFE1NEODXGzTfwiC6tXTLMQskn1X4/Rjlh0MQUvx9W+L9long==",
|
||||
"dev": true,
|
||||
"dependencies": {
|
||||
"undici-types": "~5.26.4"
|
||||
@ -1034,6 +1475,18 @@
|
||||
"wrap-ansi": "^7.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/color": {
|
||||
"version": "4.2.3",
|
||||
"resolved": "https://registry.npmjs.org/color/-/color-4.2.3.tgz",
|
||||
"integrity": "sha512-1rXeuUUiGGrykh+CeBdu5Ie7OJwinCgQY0bc7GCRxy5xVHy+moaqkpL/jqQq0MtQOeYcrqEz4abc5f0KtU7W4A==",
|
||||
"dependencies": {
|
||||
"color-convert": "^2.0.1",
|
||||
"color-string": "^1.9.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=12.5.0"
|
||||
}
|
||||
},
|
||||
"node_modules/color-convert": {
|
||||
"version": "2.0.1",
|
||||
"resolved": "https://registry.npmjs.org/color-convert/-/color-convert-2.0.1.tgz",
|
||||
@ -1050,6 +1503,15 @@
|
||||
"resolved": "https://registry.npmjs.org/color-name/-/color-name-1.1.4.tgz",
|
||||
"integrity": "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA=="
|
||||
},
|
||||
"node_modules/color-string": {
|
||||
"version": "1.9.1",
|
||||
"resolved": "https://registry.npmjs.org/color-string/-/color-string-1.9.1.tgz",
|
||||
"integrity": "sha512-shrVawQFojnZv6xM40anx4CkoDP+fZsw/ZerEMsW/pyzsRbElpsL/DBVW7q3ExxwusdNXI3lXpuhEZkzs8p5Eg==",
|
||||
"dependencies": {
|
||||
"color-name": "^1.0.0",
|
||||
"simple-swizzle": "^0.2.2"
|
||||
}
|
||||
},
|
||||
"node_modules/combined-stream": {
|
||||
"version": "1.0.8",
|
||||
"resolved": "https://registry.npmjs.org/combined-stream/-/combined-stream-1.0.8.tgz",
|
||||
@ -1244,6 +1706,14 @@
|
||||
"node": ">= 0.8"
|
||||
}
|
||||
},
|
||||
"node_modules/detect-libc": {
|
||||
"version": "2.0.2",
|
||||
"resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.0.2.tgz",
|
||||
"integrity": "sha512-UX6sGumvvqSaXgdKGUsgZWqcUyIXZ/vZTrlRT/iobiKhGL0zL4d3osHj3uqllWJK+i+sixDS/3COVEOFbupFyw==",
|
||||
"engines": {
|
||||
"node": ">=8"
|
||||
}
|
||||
},
|
||||
"node_modules/dir-glob": {
|
||||
"version": "3.0.1",
|
||||
"resolved": "https://registry.npmjs.org/dir-glob/-/dir-glob-3.0.1.tgz",
|
||||
@ -2141,6 +2611,11 @@
|
||||
"node": ">= 0.10"
|
||||
}
|
||||
},
|
||||
"node_modules/is-arrayish": {
|
||||
"version": "0.3.2",
|
||||
"resolved": "https://registry.npmjs.org/is-arrayish/-/is-arrayish-0.3.2.tgz",
|
||||
"integrity": "sha512-eVRqCvVlZbuw3GrM63ovNSNAeA1K16kaR/LRY/92w0zxQ5/1YzwblUX652i4Xs9RwAGjW9d9y6X88t8OaAJfWQ=="
|
||||
},
|
||||
"node_modules/is-binary-path": {
|
||||
"version": "2.1.0",
|
||||
"resolved": "https://registry.npmjs.org/is-binary-path/-/is-binary-path-2.1.0.tgz",
|
||||
@ -3083,6 +3558,45 @@
|
||||
"resolved": "https://registry.npmjs.org/setprototypeof/-/setprototypeof-1.2.0.tgz",
|
||||
"integrity": "sha512-E5LDX7Wrp85Kil5bhZv46j8jOeboKq5JMmYM3gVGdGH8xFpPWXUMsNrlODCrkoxMEeNi/XZIwuRvY4XNwYMJpw=="
|
||||
},
|
||||
"node_modules/sharp": {
|
||||
"version": "0.33.2",
|
||||
"resolved": "https://registry.npmjs.org/sharp/-/sharp-0.33.2.tgz",
|
||||
"integrity": "sha512-WlYOPyyPDiiM07j/UO+E720ju6gtNtHjEGg5vovUk1Lgxyjm2LFO+37Nt/UI3MMh2l6hxTWQWi7qk3cXJTutcQ==",
|
||||
"hasInstallScript": true,
|
||||
"dependencies": {
|
||||
"color": "^4.2.3",
|
||||
"detect-libc": "^2.0.2",
|
||||
"semver": "^7.5.4"
|
||||
},
|
||||
"engines": {
|
||||
"libvips": ">=8.15.1",
|
||||
"node": "^18.17.0 || ^20.3.0 || >=21.0.0"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://opencollective.com/libvips"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"@img/sharp-darwin-arm64": "0.33.2",
|
||||
"@img/sharp-darwin-x64": "0.33.2",
|
||||
"@img/sharp-libvips-darwin-arm64": "1.0.1",
|
||||
"@img/sharp-libvips-darwin-x64": "1.0.1",
|
||||
"@img/sharp-libvips-linux-arm": "1.0.1",
|
||||
"@img/sharp-libvips-linux-arm64": "1.0.1",
|
||||
"@img/sharp-libvips-linux-s390x": "1.0.1",
|
||||
"@img/sharp-libvips-linux-x64": "1.0.1",
|
||||
"@img/sharp-libvips-linuxmusl-arm64": "1.0.1",
|
||||
"@img/sharp-libvips-linuxmusl-x64": "1.0.1",
|
||||
"@img/sharp-linux-arm": "0.33.2",
|
||||
"@img/sharp-linux-arm64": "0.33.2",
|
||||
"@img/sharp-linux-s390x": "0.33.2",
|
||||
"@img/sharp-linux-x64": "0.33.2",
|
||||
"@img/sharp-linuxmusl-arm64": "0.33.2",
|
||||
"@img/sharp-linuxmusl-x64": "0.33.2",
|
||||
"@img/sharp-wasm32": "0.33.2",
|
||||
"@img/sharp-win32-ia32": "0.33.2",
|
||||
"@img/sharp-win32-x64": "0.33.2"
|
||||
}
|
||||
},
|
||||
"node_modules/shebang-command": {
|
||||
"version": "2.0.0",
|
||||
"resolved": "https://registry.npmjs.org/shebang-command/-/shebang-command-2.0.0.tgz",
|
||||
@ -3104,6 +3618,14 @@
|
||||
"node": ">=8"
|
||||
}
|
||||
},
|
||||
"node_modules/simple-swizzle": {
|
||||
"version": "0.2.2",
|
||||
"resolved": "https://registry.npmjs.org/simple-swizzle/-/simple-swizzle-0.2.2.tgz",
|
||||
"integrity": "sha512-JA//kQgZtbuY83m+xT+tXJkmJncGMTFT+C+g2h2R9uxkYIrE2yy9sgmcLhCnw57/WSD+Eh3J97FPEDFnbXnDUg==",
|
||||
"dependencies": {
|
||||
"is-arrayish": "^0.3.1"
|
||||
}
|
||||
},
|
||||
"node_modules/slash": {
|
||||
"version": "3.0.0",
|
||||
"resolved": "https://registry.npmjs.org/slash/-/slash-3.0.0.tgz",
|
||||
@ -3362,6 +3884,12 @@
|
||||
"typescript": "*"
|
||||
}
|
||||
},
|
||||
"node_modules/tslib": {
|
||||
"version": "2.6.2",
|
||||
"resolved": "https://registry.npmjs.org/tslib/-/tslib-2.6.2.tgz",
|
||||
"integrity": "sha512-AEYxH93jGFPn/a2iVAwW87VuUIkR1FVUKB77NwMF7nBTDkDrrT/Hpt/IrCJ0QXhW27jTBDcf5ZY7w6RiqTMw2Q==",
|
||||
"optional": true
|
||||
},
|
||||
"node_modules/type-check": {
|
||||
"version": "0.4.0",
|
||||
"resolved": "https://registry.npmjs.org/type-check/-/type-check-0.4.0.tgz",
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "txtdot",
|
||||
"version": "1.6.1",
|
||||
"version": "1.7.0",
|
||||
"private": true,
|
||||
"description": "txtdot is an HTTP proxy that parses only text, links and pictures from pages reducing internet bandwidth usage, removing ads and heavy scripts",
|
||||
"main": "dist/app.js",
|
||||
@ -20,14 +20,15 @@
|
||||
"json-schema-to-ts": "^3.0.0",
|
||||
"linkedom": "^0.16.8",
|
||||
"micromatch": "^4.0.5",
|
||||
"route-parser": "^0.0.5"
|
||||
"route-parser": "^0.0.5",
|
||||
"sharp": "^0.33.2"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/dompurify": "^3.0.5",
|
||||
"@types/ejs": "^3.1.5",
|
||||
"@types/jsdom": "^21.1.6",
|
||||
"@types/micromatch": "^4.0.6",
|
||||
"@types/node": "^20.11.20",
|
||||
"@types/node": "^20.11.24",
|
||||
"@types/route-parser": "^0.1.7",
|
||||
"@typescript-eslint/eslint-plugin": "^7.1.0",
|
||||
"@typescript-eslint/parser": "^7.1.0",
|
||||
|
||||
16
src/app.ts
16
src/app.ts
@ -18,6 +18,9 @@ import errorHandler from './errors/handler';
|
||||
import getConfig from './config/main';
|
||||
import redirectRoute from './routes/browser/redirect';
|
||||
|
||||
import dynConfig from './config/dynamic.config';
|
||||
import configurationRoute from './routes/browser/configuration';
|
||||
|
||||
class App {
|
||||
async init() {
|
||||
const config = getConfig();
|
||||
@ -42,6 +45,7 @@ class App {
|
||||
});
|
||||
|
||||
if (config.swagger) {
|
||||
dynConfig.addRoute('/doc');
|
||||
await fastify.register(fastifySwagger, {
|
||||
swagger: {
|
||||
info: {
|
||||
@ -54,14 +58,16 @@ class App {
|
||||
await fastify.register(fastifySwaggerUi, { routePrefix: '/doc' });
|
||||
}
|
||||
|
||||
fastify.addHook('onRoute', (route) => {
|
||||
dynConfig.addRoute(route.url);
|
||||
});
|
||||
|
||||
fastify.register(indexRoute);
|
||||
fastify.register(getRoute);
|
||||
fastify.register(configurationRoute);
|
||||
|
||||
if (config.search.enabled) {
|
||||
fastify.register(redirectRoute);
|
||||
}
|
||||
|
||||
if (config.proxy_res) fastify.register(proxyRoute);
|
||||
config.search.enabled && fastify.register(redirectRoute);
|
||||
config.proxy.enabled && fastify.register(proxyRoute);
|
||||
|
||||
fastify.register(parseRoute);
|
||||
fastify.register(rawHtml);
|
||||
|
||||
@ -5,7 +5,7 @@ export class ConfigService {
|
||||
public readonly port: number;
|
||||
public readonly timeout: number;
|
||||
public readonly reverse_proxy: boolean;
|
||||
public readonly proxy_res: boolean;
|
||||
public readonly proxy: ProxyConfig;
|
||||
public readonly swagger: boolean;
|
||||
public readonly search: SearchConfig;
|
||||
|
||||
@ -19,7 +19,11 @@ export class ConfigService {
|
||||
|
||||
this.reverse_proxy = this.parseBool(process.env.REVERSE_PROXY, false);
|
||||
|
||||
this.proxy_res = this.parseBool(process.env.PROXY_RES, true);
|
||||
this.proxy = {
|
||||
enabled: this.parseBool(process.env.PROXY_RES, true),
|
||||
img_compress: this.parseBool(process.env.IMG_COMPRESS, true),
|
||||
};
|
||||
|
||||
this.swagger = this.parseBool(process.env.SWAGGER, false);
|
||||
|
||||
this.search = {
|
||||
@ -34,6 +38,11 @@ export class ConfigService {
|
||||
}
|
||||
}
|
||||
|
||||
interface ProxyConfig {
|
||||
enabled: boolean;
|
||||
img_compress: boolean;
|
||||
}
|
||||
|
||||
interface SearchConfig {
|
||||
enabled: boolean;
|
||||
searx_url?: string;
|
||||
|
||||
10
src/config/dynamic.config.ts
Normal file
10
src/config/dynamic.config.ts
Normal file
@ -0,0 +1,10 @@
|
||||
class DynConfigService {
|
||||
public routes: Set<string> = new Set();
|
||||
constructor() {}
|
||||
addRoute(route: string) {
|
||||
this.routes.add(route);
|
||||
}
|
||||
}
|
||||
|
||||
const config = new DynConfigService();
|
||||
export default config;
|
||||
@ -55,7 +55,8 @@ function htmlErrorHandler(error: Error, reply: FastifyReply, url: string) {
|
||||
url,
|
||||
code: error.code,
|
||||
description: error.description,
|
||||
proxyBtn: error instanceof NotHtmlMimetypeError && getConfig().proxy_res,
|
||||
proxyBtn:
|
||||
error instanceof NotHtmlMimetypeError && getConfig().proxy.enabled,
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
@ -31,13 +31,23 @@ export class LocalResourceError extends TxtDotError {
|
||||
}
|
||||
}
|
||||
|
||||
export class UnsupportedMimetypeError extends TxtDotError {
|
||||
constructor(expected: string, got?: string) {
|
||||
super(
|
||||
415,
|
||||
'UnsupportedMimetypeError',
|
||||
`Unsupported mimetype, expected ${expected}, got ${got}`
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
export class NotHtmlMimetypeError extends TxtDotError {
|
||||
constructor() {
|
||||
super(
|
||||
421,
|
||||
'NotHtmlMimetypeError',
|
||||
'Received non-HTML content, ' +
|
||||
(getConfig().proxy_res
|
||||
(getConfig().proxy.enabled
|
||||
? 'use proxy instead of parser.'
|
||||
: 'proxying is disabled by the instance admin.')
|
||||
);
|
||||
|
||||
@ -7,9 +7,7 @@ import DOMPurify from 'dompurify';
|
||||
|
||||
import { Readable } from 'stream';
|
||||
|
||||
import isLocalResource from '../utils/islocal';
|
||||
|
||||
import { LocalResourceError, NotHtmlMimetypeError } from '../errors/main';
|
||||
import { NotHtmlMimetypeError } from '../errors/main';
|
||||
import { HandlerInput } from './handler-input';
|
||||
import { decodeStream, parseEncodingName } from '../utils/http';
|
||||
import replaceHref from '../utils/replace-href';
|
||||
@ -40,10 +38,6 @@ export class Distributor {
|
||||
): Promise<IHandlerOutput> {
|
||||
const urlObj = new URL(remoteUrl);
|
||||
|
||||
if (await isLocalResource(urlObj)) {
|
||||
throw new LocalResourceError();
|
||||
}
|
||||
|
||||
const response = await axios.get(remoteUrl);
|
||||
const data: Readable = response.data;
|
||||
const mime: string | undefined =
|
||||
@ -76,6 +70,7 @@ export class Distributor {
|
||||
if (specified) {
|
||||
return this.fallback[this.engines_id[specified]];
|
||||
}
|
||||
|
||||
for (const engine of this.fallback) {
|
||||
if (micromatch.isMatch(host, engine.domains)) {
|
||||
return engine;
|
||||
|
||||
@ -11,12 +11,14 @@ interface IRoute<TParams extends RouteValues> {
|
||||
|
||||
export class Engine {
|
||||
name: string;
|
||||
description: string;
|
||||
domains: string[];
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
routes: IRoute<any>[] = [];
|
||||
constructor(name: string, domains: string[] = []) {
|
||||
constructor(name: string, description: string, domains: string[] = []) {
|
||||
this.domains = domains;
|
||||
this.name = name;
|
||||
this.description = description;
|
||||
}
|
||||
|
||||
route<TParams extends RouteValues>(
|
||||
|
||||
@ -3,7 +3,11 @@ import { EngineParseError } from '../../errors/main';
|
||||
|
||||
import { Engine } from '../engine';
|
||||
|
||||
const ReadabilityEngine = new Engine('Readability');
|
||||
const ReadabilityEngine = new Engine(
|
||||
'Readability',
|
||||
'Engine for parsing content with Readability',
|
||||
['*']
|
||||
);
|
||||
|
||||
ReadabilityEngine.route('*path', async (input, ro) => {
|
||||
const reader = new Readability(input.parseDom().window.document);
|
||||
|
||||
@ -2,7 +2,9 @@ import { Route } from '../../types/handlers';
|
||||
import { Engine } from '../engine';
|
||||
import { HandlerInput } from '../handler-input';
|
||||
|
||||
const SearXEngine = new Engine('SearX', ['searx.*']);
|
||||
const SearXEngine = new Engine('SearX', "Engine for searching with 'SearXNG'", [
|
||||
'searx.*',
|
||||
]);
|
||||
|
||||
async function search(
|
||||
input: HandlerInput,
|
||||
|
||||
@ -1,16 +1,20 @@
|
||||
import { Engine } from '../../engine';
|
||||
import questions from './questions';
|
||||
import users from './users';
|
||||
const soEngine = new Engine('StackOverflow', [
|
||||
'stackoverflow.com',
|
||||
'*.stackoverflow.com',
|
||||
'*.stackexchange.com',
|
||||
'askubuntu.com',
|
||||
'stackapps.com',
|
||||
'mathoverflow.net',
|
||||
'superuser.com',
|
||||
'serverfault.com',
|
||||
]);
|
||||
const soEngine = new Engine(
|
||||
'StackOverflow',
|
||||
"Engine for 'StackOverflow'. Available routes: '/questions/' and '/users/'",
|
||||
[
|
||||
'stackoverflow.com',
|
||||
'*.stackoverflow.com',
|
||||
'*.stackexchange.com',
|
||||
'askubuntu.com',
|
||||
'stackapps.com',
|
||||
'mathoverflow.net',
|
||||
'superuser.com',
|
||||
'serverfault.com',
|
||||
]
|
||||
);
|
||||
|
||||
soEngine.route('/questions/:id/*slug', questions);
|
||||
soEngine.route('/users/:id/*slug', users);
|
||||
|
||||
@ -5,9 +5,9 @@ import StackOverflow from './engines/stackoverflow/main';
|
||||
|
||||
const distributor = new Distributor();
|
||||
|
||||
distributor.engine(Readability);
|
||||
distributor.engine(SearX);
|
||||
distributor.engine(StackOverflow);
|
||||
distributor.engine(SearX);
|
||||
distributor.engine(Readability);
|
||||
|
||||
export const engineList = distributor.list;
|
||||
export default distributor;
|
||||
|
||||
19
src/routes/browser/configuration.ts
Normal file
19
src/routes/browser/configuration.ts
Normal file
@ -0,0 +1,19 @@
|
||||
import { FastifyInstance } from 'fastify';
|
||||
|
||||
import packageJSON from '../../package';
|
||||
import distributor from '../../handlers/main';
|
||||
import { indexSchema } from '../../types/requests/browser';
|
||||
|
||||
import getConfig from '../../config/main';
|
||||
import dynConfig from '../../config/dynamic.config';
|
||||
|
||||
export default async function configurationRoute(fastify: FastifyInstance) {
|
||||
fastify.get('/configuration', { schema: indexSchema }, async (_, reply) => {
|
||||
return reply.view('/templates/configuration.ejs', {
|
||||
packageJSON,
|
||||
engines: distributor.fallback,
|
||||
dynConfig,
|
||||
config: getConfig(),
|
||||
});
|
||||
});
|
||||
}
|
||||
@ -1,6 +1,9 @@
|
||||
import { FastifyInstance } from 'fastify';
|
||||
import { IProxySchema, ProxySchema } from '../../types/requests/browser';
|
||||
import axios from '../../types/axios';
|
||||
import sharp from 'sharp';
|
||||
import getConfig from '../../config/main';
|
||||
import { UnsupportedMimetypeError } from '../../errors/main';
|
||||
|
||||
import isLocalResource from '../../utils/islocal';
|
||||
import { LocalResourceError } from '../../errors/main';
|
||||
@ -24,4 +27,49 @@ export default async function proxyRoute(fastify: FastifyInstance) {
|
||||
return reply.send(response.data);
|
||||
}
|
||||
);
|
||||
|
||||
if (getConfig().proxy.img_compress)
|
||||
fastify.get<IProxySchema>(
|
||||
'/proxy/img',
|
||||
{ schema: ProxySchema },
|
||||
async (request, reply) => {
|
||||
const response = await axios.get(request.query.url, {
|
||||
responseType: 'arraybuffer',
|
||||
});
|
||||
|
||||
const mime: string | undefined =
|
||||
response.headers['content-type']?.toString();
|
||||
|
||||
if (!(mime && mime.startsWith('image/'))) {
|
||||
throw new UnsupportedMimetypeError('image/*', mime);
|
||||
}
|
||||
|
||||
const clen: number | undefined = parseInt(
|
||||
response.headers['content-length']?.toString() || '0'
|
||||
);
|
||||
|
||||
if (mime.startsWith('image/svg')) {
|
||||
reply.header('Content-Type', mime);
|
||||
reply.header('Content-Length', clen);
|
||||
return reply.send(response.data);
|
||||
}
|
||||
|
||||
const buffer = await sharp(response.data)
|
||||
// .grayscale(true)
|
||||
.toFormat('webp', {
|
||||
quality: 25,
|
||||
progressive: true,
|
||||
optimizeScans: true,
|
||||
})
|
||||
.toBuffer();
|
||||
|
||||
reply.header('Content-Type', 'image/webp');
|
||||
reply.header('Content-Length', buffer.length);
|
||||
|
||||
reply.header('x-original-size', clen);
|
||||
reply.header('x-bytes-saved', clen - buffer.length);
|
||||
|
||||
return reply.send(buffer);
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
@ -1,9 +1,30 @@
|
||||
import axios from 'axios';
|
||||
import origAxios from 'axios';
|
||||
import { isLocalResource, isLocalResourceURL } from '../utils/islocal';
|
||||
import { LocalResourceError } from '../errors/main';
|
||||
|
||||
export default axios.create({
|
||||
const axios = origAxios.create({
|
||||
headers: {
|
||||
'User-Agent':
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/116.0',
|
||||
},
|
||||
responseType: 'stream',
|
||||
});
|
||||
|
||||
axios.interceptors.response.use(
|
||||
(response) => {
|
||||
if (isLocalResource(response.request.socket.remoteAddress)) {
|
||||
throw new LocalResourceError();
|
||||
}
|
||||
|
||||
return response;
|
||||
},
|
||||
async (error) => {
|
||||
if (await isLocalResourceURL(new URL(error.config?.url))) {
|
||||
throw new LocalResourceError();
|
||||
}
|
||||
|
||||
throw error;
|
||||
}
|
||||
);
|
||||
|
||||
export default axios;
|
||||
|
||||
@ -27,12 +27,13 @@ export function generateParserUrl(
|
||||
export function generateProxyUrl(
|
||||
requestUrl: URL,
|
||||
remoteUrl: URL,
|
||||
href: string
|
||||
href: string,
|
||||
subProxy?: string
|
||||
): string {
|
||||
const realHref = getRealURL(href, remoteUrl);
|
||||
|
||||
const urlParam = `?url=${encodeURIComponent(realHref.href)}`;
|
||||
return `${requestUrl.origin}/proxy${urlParam}`;
|
||||
return `${requestUrl.origin}/proxy${subProxy ? `/${subProxy}` : ''}${urlParam}`;
|
||||
}
|
||||
|
||||
function getRealURL(href: string, remoteUrl: URL) {
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
import dns from 'dns';
|
||||
import ipRangeCheck from 'ip-range-check';
|
||||
import dns from 'dns';
|
||||
|
||||
const subnets = [
|
||||
'0.0.0.0/8',
|
||||
@ -35,7 +35,11 @@ const subnets = [
|
||||
'ff00::/8',
|
||||
];
|
||||
|
||||
export default async function isLocalResource(url: URL): Promise<boolean> {
|
||||
export function isLocalResource(addr: string): boolean {
|
||||
return ipRangeCheck(addr, subnets);
|
||||
}
|
||||
|
||||
export async function isLocalResourceURL(url: URL): Promise<boolean> {
|
||||
// Resolve domain name
|
||||
const addr = (await dns.promises.lookup(url.hostname)).address;
|
||||
|
||||
|
||||
@ -15,16 +15,27 @@ export default function replaceHref(
|
||||
const proxyUrl = (href: string) =>
|
||||
generateProxyUrl(requestUrl, remoteUrl, href);
|
||||
|
||||
const imgProxyUrl = (href: string) =>
|
||||
generateProxyUrl(requestUrl, remoteUrl, href, 'img');
|
||||
|
||||
modifyLinks(doc.querySelectorAll('a[href]'), 'href', parserUrl);
|
||||
modifyLinks(doc.querySelectorAll('frame,iframe'), 'src', parserUrl);
|
||||
|
||||
if (getConfig().proxy_res) {
|
||||
const config = getConfig();
|
||||
|
||||
if (config.proxy.enabled) {
|
||||
modifyLinks(
|
||||
doc.querySelectorAll('img,image,video,audio,embed,track,source'),
|
||||
doc.querySelectorAll('video,audio,embed,track,source'),
|
||||
'src',
|
||||
proxyUrl
|
||||
);
|
||||
|
||||
modifyLinks(
|
||||
doc.querySelectorAll('img,image'),
|
||||
'src',
|
||||
config.proxy.img_compress ? imgProxyUrl : proxyUrl
|
||||
);
|
||||
|
||||
modifyLinks(doc.getElementsByTagName('object'), 'data', proxyUrl);
|
||||
const sources = doc.querySelectorAll('source,img');
|
||||
for (const source of sources) {
|
||||
|
||||
26
static/configuration.css
Normal file
26
static/configuration.css
Normal file
@ -0,0 +1,26 @@
|
||||
main {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
align-items: start;
|
||||
gap: 0.375rem;
|
||||
}
|
||||
|
||||
h1 {
|
||||
width: fit-content;
|
||||
margin: auto;
|
||||
}
|
||||
|
||||
h1 > .dot {
|
||||
color: var(--accent);
|
||||
}
|
||||
|
||||
.menu {
|
||||
justify-content: center;
|
||||
}
|
||||
|
||||
.configuration {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
align-items: start;
|
||||
gap: 0.375rem;
|
||||
}
|
||||
@ -1,26 +1,30 @@
|
||||
<% search = config.search.enabled %>
|
||||
|
||||
<% if (search) { %>
|
||||
<%
|
||||
|
||||
<input type="checkbox" id="switch-search" checked>
|
||||
if (search) {
|
||||
%>
|
||||
<input type="checkbox" id="switch-search" checked>
|
||||
|
||||
<label for="switch-search" class="switch-label">
|
||||
<span>URL</span>
|
||||
<span class="switch-btn"></span>
|
||||
<span>Search</span>
|
||||
</label>
|
||||
<label for="switch-search" class="switch-label">
|
||||
<span>URL</span>
|
||||
<span class="switch-btn"></span>
|
||||
<span>Search</span>
|
||||
</label>
|
||||
|
||||
<form action="/redirect" method="get" class="input-grid main-form-search">
|
||||
<div class="input">
|
||||
<input type="text" name="q" id="search" placeholder="Search">
|
||||
</div>
|
||||
<div class="input">
|
||||
<input type="submit" id="submit" class="button" value="Go">
|
||||
</div>
|
||||
<input type="hidden" name="url" value="<%= config.search.searx_url %>/search"/>
|
||||
</form>
|
||||
<form action="/redirect" method="get" class="input-grid main-form-search">
|
||||
<div class="input">
|
||||
<input type="text" name="q" id="search" placeholder="Search">
|
||||
</div>
|
||||
<div class="input">
|
||||
<input type="submit" id="submit" class="button" value="Go">
|
||||
</div>
|
||||
<input type="hidden" name="url" value="<%= config.search.searx_url %>/search"/>
|
||||
</form>
|
||||
<%
|
||||
}
|
||||
|
||||
<% } %>
|
||||
%>
|
||||
|
||||
<form action="/get" method="get" class="input-grid <%= search ? "main-form-url" : "" %>">
|
||||
<div class="input">
|
||||
|
||||
54
templates/configuration.ejs
Normal file
54
templates/configuration.ejs
Normal file
@ -0,0 +1,54 @@
|
||||
<%
|
||||
|
||||
// hide private properties from config
|
||||
const to_hide = ["host", "port"];
|
||||
function replacer(key,value) {
|
||||
if (to_hide.includes(key)) return undefined;
|
||||
else return value;
|
||||
}
|
||||
function to_pretty(obj) {
|
||||
return JSON.stringify(obj, replacer, 2).replace(/[\[\]{}"]/g, "").replace(/,/g, "");
|
||||
}
|
||||
%>
|
||||
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<meta http-equiv="X-UA-Compatible" content="IE=edge">
|
||||
<meta name="description" content="<%= packageJSON.description %>">
|
||||
<title>txt. configuration</title>
|
||||
<link rel="stylesheet" href="/static/common.css">
|
||||
<link rel="stylesheet" href="/static/configuration.css">
|
||||
</head>
|
||||
<body>
|
||||
<main>
|
||||
<header>
|
||||
<h1>txt<span class="dot">.</span></h1>
|
||||
<div class="menu">
|
||||
<a class="button secondary" href="/">Home</a>
|
||||
</div>
|
||||
<p><%= packageJSON.description %></p>
|
||||
</header>
|
||||
<div class="configuration">
|
||||
<h2>Configuration</h2>
|
||||
<pre> version: <%= packageJSON.version %><%= to_pretty(config) %></pre>
|
||||
<h2>Available engines</h2>
|
||||
<ol>
|
||||
<%
|
||||
for (const engine of engines) {
|
||||
%><li><%= engine.name %>: <%= engine.description %></li><%
|
||||
}
|
||||
%>
|
||||
</ol>
|
||||
<h2>Available routes</h2>
|
||||
<%
|
||||
for (const route of dynConfig.routes) {
|
||||
%><a class="button secondary" href="<%= route %>"><%= route %></a><%
|
||||
}
|
||||
%>
|
||||
</div>
|
||||
</main>
|
||||
</body>
|
||||
</html>
|
||||
Loading…
x
Reference in New Issue
Block a user