Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New Self-hosted Version Progress & issues #5069

Closed
7 of 8 tasks
EYHN opened this issue Nov 27, 2023 · 48 comments
Closed
7 of 8 tasks

New Self-hosted Version Progress & issues #5069

EYHN opened this issue Nov 27, 2023 · 48 comments
Labels
type:epic Big feature type:feat Non-bug feature improvements

Comments

@EYHN
Copy link
Member

EYHN commented Nov 27, 2023

UPDATED: selfhost has been released! Please refer to our selfhost tutorial. https://docs.affine.pro/docs/self-host-affine

If you have tried our docker before, don't forget to pull the image again.

If you encounter any problems, please open a new issue.

Many people are interested in the development of our new self-hosted version, so I created this issue to keep everyone updated on its progress.

The current version of docker image has lagged behind our stable branch for a long time. We need to update the new docker-image.

Since we ship many features like payment and account on the current backend server, it is not suitable for common self-hosted use cases. So we need to make these features optional in the new self-hosted version, which requires some work.

Tasks

  1. darkskygit
  2. app:server mod:payment
    darkskygit
  3. roadmap story type:feat
  4. app:core app:server mod:dev mod:workspace-impl
    Brooooooklyn
  5. app:server
    forehalo
  6. app:core mod:component mod:i18n test
    forehalo

Archived issues

@omarcostahamido
Copy link

🏆 a self-host tutorial on synology would be golden 🙌

@PhilippMolitor
Copy link

@omarcostahamido I think there is endless potential for integration into other managed systems.
A few integrations that would enable more non-tech-people (or those who know about local systems, but nothing about application development/deployment) to use AFFiNE:

  • CasaOS
  • Synology DSM
  • QNAP QTS
  • Unraid

@omarcostahamido
Copy link

omarcostahamido commented Dec 28, 2023

@PhilippMolitor

Yes synology DSM, that's what I meant. I'm happy to help with it!

@bphenriques
Copy link

I have a synology and I prefer have a docker compose setup

@mmospanenko
Copy link

Waiting for official Docker image (ARM64). This is required for any server-side projects nowadays :)

@thelazyoxymoron
Copy link

Looks like the docker build for selfhost support got merged recently. Excited to try this!

@TheNexter
Copy link

Waiting for official Docker image (ARM64). This is required for any server-side projects nowadays :)

The new version drop few hours ago can be run on ARM64 using a raspberry pi 4 😄

@TheNexter
Copy link

The new version for self hosted is running but i have this message :
image

My server not gonna save my note ? or it's a UI bug ?

@Exitialis
Copy link

Exitialis commented Feb 2, 2024

I tried to run this docker-compose on my server too, but i faced with several issues:

  1. Problem with docker-compose version - in the compose.yaml version tag didnt mentionted and because of that i faced with error "Unsupported config option for services: 'postgres'", when i added version: "3" at the top of the file, this error gone.
  2. The next problem is that in the latest docker-compose, depends-on tag have different type and should be described differently
  3. After all fixes for docker-compose, everything was up, but i found, that only frontend version was launched and didnt save docs to the database and in no option in desktop version to configurate my server url in order to use it as cloud, only one option - activate affine cloud
  4. The last problem - i didnt find how to setup OAuth to provide login through google, and button from interface just redirect me back on frontend part login form with token, but login was not completed

I can provide any help that you need with self-hosted version, even write instructions on how to run affine on cloud-ocean droplet or drop docker image on TrueNas Scale.

@omarcostahamido
Copy link

omarcostahamido commented Feb 2, 2024

@Exitialis Thank you for sharing your experience and the workarounds that you found!

no option in desktop version to configurate my server url in order to use it as cloud

I guess that's why #2646 is open?

@fourdim
Copy link
Contributor

fourdim commented Feb 14, 2024

The new version for self hosted is running but i have this message : image

My server not gonna save my note ? or it's a UI bug ?

The server is not gonna save you note unless the cloud is enabled. All data will be saved in the browser's local storage.
It can be easily erased because of cleaning browsing data.

@benjick
Copy link

benjick commented Feb 22, 2024

I tried running the self-host docker-compose myself but whatever should run on port 5555 isn't starting

@Brooooooklyn
Copy link
Member

I tried running the self-host docker-compose myself but whatever should run on port 5555 isn't starting

5555 is reserved for Primsa Studio, you can run yarn prisma studio to start Primsa Studio.

@benjick
Copy link

benjick commented Feb 22, 2024

I see, thank you @Brooooooklyn!

Then I have some other issue, if I go to http://123.123.123.123:3010 I just see this:

{"compatibility":"0.12.0","message":"AFFiNE 0.12.0 Server","flavor":{"type":"allinone","main":true,"sync":true,"selfhosted":false}}

I can see the logs saying this:

And the public server should be recognized as https://affine.example.com

But If browse to https://affine.example.com I get a cloudflare timeout. What are the steps to troubleshoot here?

@simonkaiser9
Copy link

@benjick I had the same issue, which stemmed from me using the docker-compose.yaml from the canary-branch instead of 0.12.0. On canary SERVER_FLAVOR is not set, compared to 0.12.0 where the following is added in the docker-compose file:

  • SERVER_FLAVOR=selfhosted

Although my service now comes up as expected and my redis and graph db are both up and running and configured, I still get the prompt, that my data is only stored in the browser instead of in the database:
I cannot find an option to active the db storage explicitly and wonder, what may go wrong during the start up of the service...
image

@acranglesim
Copy link

I see the same issue as above, when I try to sign in - the frontend says "GraphQL empty body"

And the docker log:

[Nest] 113  - 02/23/2024, 10:37:17 AM    WARN [NextAuth] NEXTAUTH_URL

[Nest] 113  - 02/23/2024, 10:37:46 AM   ERROR [ExceptionLogger] Error: requestId-HajWAl7Na5GhwDcaPdLD-: Cannot POST /graphql

@simonkaiser9
Copy link

The new version for self hosted is running but i have this message : image
My server not gonna save my note ? or it's a UI bug ?

The server is not gonna save you note unless the cloud is enabled. All data will be saved in the browser's local storage. It can be easily erased because of cleaning browsing data.

@fourdim What is the GraphDB used for then? My understanding was, that we'd be able to save the data to the locally run graph db service.

@acranglesim
Copy link

As an addendum, I get the same error listed above after trying out the :stable version too.

@EYHN
Copy link
Member Author

EYHN commented Feb 26, 2024

@acranglesim @simonkaiser9

Sorry, our self-hosted version is not yet ready, as it seems that the configuration module is not working properly. Since it is already outside of working hours in our engineers' time zone, I believe we will resolve this issue tomorrow.

@panw3i
Copy link

panw3i commented Feb 27, 2024

Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/app/scripts/register.js' imported from /app/


> > FYI, just noticed this after the announcement today. There is a docs.affine.pro page that has a self-host tutorial now: https://docs.affine.pro/docs/self-host-affine-
> 
> I get node errors using the provided unmodified self-hosted compose.yaml
> 
> ```
> affine_selfhosted  | node:internal/errors:496
> affine_selfhosted  |     ErrorCaptureStackTrace(err);
> affine_selfhosted  |     ^
> affine_selfhosted  |
> affine_selfhosted  | Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/app/scripts/register.js' imported from /app/
> affine_selfhosted  |     at new NodeError (node:internal/errors:405:5)
> affine_selfhosted  |     at finalizeResolution (node:internal/modules/esm/resolve:327:11)
> affine_selfhosted  |     at moduleResolve (node:internal/modules/esm/resolve:980:10)
> affine_selfhosted  |     at defaultResolve (node:internal/modules/esm/resolve:1193:11)
> affine_selfhosted  |     at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:403:12)
> affine_selfhosted  |     at ModuleLoader.resolve (node:internal/modules/esm/loader:372:25)
> affine_selfhosted  |     at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:249:38)
> affine_selfhosted  |     at ModuleLoader.import (node:internal/modules/esm/loader:335:34)
> affine_selfhosted  |     at node:internal/process/esm_loader:26:84
> affine_selfhosted  |     at node:internal/per_context/primordials:544:39 {
> affine_selfhosted  |   url: 'file:///app/scripts/register.js',
> affine_selfhosted  |   code: 'ERR_MODULE_NOT_FOUND'
> affine_selfhosted  | }
> affine_selfhosted  |
> affine_selfhosted  | Node.js v18.19.1
> 

FYI, just noticed this after the announcement today. There is a docs.affine.pro page that has a self-host tutorial now: https://docs.affine.pro/docs/self-host-affine-

I get node errors using the provided unmodified self-hosted compose.yaml

affine_selfhosted  | node:internal/errors:496
affine_selfhosted  |     ErrorCaptureStackTrace(err);
affine_selfhosted  |     ^
affine_selfhosted  |
affine_selfhosted  | Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/app/scripts/register.js' imported from /app/
affine_selfhosted  |     at new NodeError (node:internal/errors:405:5)
affine_selfhosted  |     at finalizeResolution (node:internal/modules/esm/resolve:327:11)
affine_selfhosted  |     at moduleResolve (node:internal/modules/esm/resolve:980:10)
affine_selfhosted  |     at defaultResolve (node:internal/modules/esm/resolve:1193:11)
affine_selfhosted  |     at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:403:12)
affine_selfhosted  |     at ModuleLoader.resolve (node:internal/modules/esm/loader:372:25)
affine_selfhosted  |     at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:249:38)
affine_selfhosted  |     at ModuleLoader.import (node:internal/modules/esm/loader:335:34)
affine_selfhosted  |     at node:internal/process/esm_loader:26:84
affine_selfhosted  |     at node:internal/per_context/primordials:544:39 {
affine_selfhosted  |   url: 'file:///app/scripts/register.js',
affine_selfhosted  |   code: 'ERR_MODULE_NOT_FOUND'
affine_selfhosted  | }
affine_selfhosted  |
affine_selfhosted  | Node.js v18.19.1

@PhilippMolitor
Copy link

PhilippMolitor commented Feb 27, 2024

image

i can confirm the register.js file is missing. when changing the import in docker-compose env to the dist/prelude.js file, I get the same error as @PovilasID . when fully removing the node options, i get the same error with more stuff to it:

affine_1    | Error [ERR_UNSUPPORTED_DIR_IMPORT]: Directory import '/app/dist/fundamentals/config' is not supported resolving ES modules imported from /app/dist/prelude.js
affine_1    |     at new NodeError (node:internal/errors:405:5)
affine_1    |     at finalizeResolution (node:internal/modules/esm/resolve:321:11)
affine_1    |     at moduleResolve (node:internal/modules/esm/resolve:980:10)
affine_1    |     at defaultResolve (node:internal/modules/esm/resolve:1193:11)
affine_1    |     at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:403:12)
affine_1    |     at ModuleLoader.resolve (node:internal/modules/esm/loader:372:25)
affine_1    |     at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:249:38)
affine_1    |     at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:76:39)
affine_1    |     at link (node:internal/modules/esm/module_job:75:36) {
affine_1    |   url: 'file:///app/dist/fundamentals/config',
affine_1    |   code: 'ERR_UNSUPPORTED_DIR_IMPORT'
affine_1    | }
affine_1    |
affine_1    | Node.js v18.19.1
affine_1    |
affine_1    |     at checkExecSyncError (node:child_process:890:11)
affine_1    |     at execSync (node:child_process:962:15)
affine_1    |     at runPredeployScript (file:///app/scripts/self-host-predeploy.js:41:3)
affine_1    |     at file:///app/scripts/self-host-predeploy.js:51:1
affine_1    |     at ModuleJob.run (node:internal/modules/esm/module_job:195:25)
affine_1    |     at async ModuleLoader.import (node:internal/modules/esm/loader:336:24)
affine_1    |     at async loadESM (node:internal/process/esm_loader:34:7)
affine_1    |     at async handleMainPromise (node:internal/modules/run_main:106:12) {
affine_1    |   status: 1,
affine_1    |   signal: null,
affine_1    |   output: [
affine_1    |     null,
affine_1    |     Buffer(0) [Uint8Array] [],
affine_1    |     Buffer(932) [Uint8Array] [
affine_1    |       110, 111, 100, 101,  58, 105, 110, 116, 101, 114, 110,  97,
affine_1    |       108,  47, 101, 114, 114, 111, 114, 115,  58,  52,  57,  54,
affine_1    |        10,  32,  32,  32,  32,  69, 114, 114, 111, 114,  67,  97,
affine_1    |       112, 116, 117, 114, 101,  83, 116,  97,  99, 107,  84, 114,
affine_1    |        97,  99, 101,  40, 101, 114, 114,  41,  59,  10,  32,  32,
affine_1    |        32,  32,  94,  10,  10,  69, 114, 114, 111, 114,  32,  91,
affine_1    |        69,  82,  82,  95,  85,  78,  83,  85,  80,  80,  79,  82,
affine_1    |        84,  69,  68,  95,  68,  73,  82,  95,  73,  77,  80,  79,
affine_1    |        82,  84,  93,  58,
affine_1    |       ... 832 more items
affine_1    |     ]
affine_1    |   ],
affine_1    |   pid: 19,
affine_1    |   stdout: Buffer(0) [Uint8Array] [],
affine_1    |   stderr: Buffer(932) [Uint8Array] [
affine_1    |     110, 111, 100, 101,  58, 105, 110, 116, 101, 114, 110,  97,
affine_1    |     108,  47, 101, 114, 114, 111, 114, 115,  58,  52,  57,  54,
affine_1    |      10,  32,  32,  32,  32,  69, 114, 114, 111, 114,  67,  97,
affine_1    |     112, 116, 117, 114, 101,  83, 116,  97,  99, 107,  84, 114,
affine_1    |      97,  99, 101,  40, 101, 114, 114,  41,  59,  10,  32,  32,
affine_1    |      32,  32,  94,  10,  10,  69, 114, 114, 111, 114,  32,  91,
affine_1    |      69,  82,  82,  95,  85,  78,  83,  85,  80,  80,  79,  82,
affine_1    |      84,  69,  68,  95,  68,  73,  82,  95,  73,  77,  80,  79,
affine_1    |      82,  84,  93,  58,
affine_1    |     ... 832 more items
affine_1    |   ]
affine_1    | }

@xuqingkuang
Copy link

xuqingkuang commented Feb 27, 2024

FYI, just noticed this after the announcement today. There is a docs.affine.pro page that has a self-host tutorial now: https://docs.affine.pro/docs/self-host-affine-

I get node errors using the provided unmodified self-hosted compose.yaml

affine_selfhosted  | node:internal/errors:496
affine_selfhosted  |     ErrorCaptureStackTrace(err);
affine_selfhosted  |     ^
affine_selfhosted  |
affine_selfhosted  | Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/app/scripts/register.js' imported from /app/
affine_selfhosted  |     at new NodeError (node:internal/errors:405:5)
affine_selfhosted  |     at finalizeResolution (node:internal/modules/esm/resolve:327:11)
affine_selfhosted  |     at moduleResolve (node:internal/modules/esm/resolve:980:10)
affine_selfhosted  |     at defaultResolve (node:internal/modules/esm/resolve:1193:11)
affine_selfhosted  |     at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:403:12)
affine_selfhosted  |     at ModuleLoader.resolve (node:internal/modules/esm/loader:372:25)
affine_selfhosted  |     at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:249:38)
affine_selfhosted  |     at ModuleLoader.import (node:internal/modules/esm/loader:335:34)
affine_selfhosted  |     at node:internal/process/esm_loader:26:84
affine_selfhosted  |     at node:internal/per_context/primordials:544:39 {
affine_selfhosted  |   url: 'file:///app/scripts/register.js',
affine_selfhosted  |   code: 'ERR_MODULE_NOT_FOUND'
affine_selfhosted  | }
affine_selfhosted  |
affine_selfhosted  | Node.js v18.19.1

The PR #5920 was recently merged. The documentation has been updated, but obviously, the new version of stable tag hasn't been published yet.

You just need to restore the environment.

- - NODE_OPTIONS="--import=./scripts/register.js"
+ - NODE_OPTIONS="--es-module-specifier-resolution=node"

@xuqingkuang
Copy link

Thanks to the tip from the person above, I managed to get Affine running locally. However, please don't have too high expectations as it is still a fair way off from replacing Notion at this point.

  1. The web UI cannot sync data with self-hosted server yet, it still working with local (browser) data only.
  2. The newly added GraphQL API seems to haven't been implemented in the self-hosted server yet, it currently returns a 404 error. - It maybe Affine Cloud only.

Local only

@peterbuga
Copy link

@xuqingkuang thanks for the reply, I managed to get it up'n'running but there's a bit of inconsistency I'm missing.

by default, it spins up the graphql server on port :3010. using comment above #5069 (comment) it drops the graphql server and spins up the web UI on the same port but when I try to login is looking for :3010/graphql that's not available anymore.

in theory i can spin up 2 instances and route the requests to /graphql to server instance using traefik for example, i'll try it later on and see if this a solution 🤞

@PovilasID
Copy link

@peterbuga looks like graphql cloud be on secondary port... or two containers? But honestly it sounds like classic case of two services trying claiming the same port. Fore somebody with knowledge of the architecture should not be a very hard fix.
In mean time how do I prevent the container from trying to star web UI? I can probably work out traefik config from there....

@peterbuga
Copy link

@PovilasID I did some testing, using 2 containers and routing the /graphql requests to backend using traefik as mentioned above.
I know it's a lot of work from the devs but I do believe it's now yet ready for us to play around, for example the migrations are never ran, the admin user it's not getting created (the db is empty), there's more to it and we should just wait for a while until they get sorted out.

thanks for all the hard work! i'll keep an eye on the project

@PhilippMolitor
Copy link

For me, the app under port 3010 responds with the following at /:

{"compatibility":"0.12.0","message":"AFFiNE 0.12.0 Server","flavor":{"type":"allinone","main":true,"sync":true,"selfhosted":false}}

The /graphql path responds with the GraphQL Playground UI.

@acranglesim
Copy link

@PhilippMolitor you need to add:
SERVER_FLAVOR=selfhosted to your docker-compile file in the environment section of affine

@fish-uncle
Copy link

/graphql is 404
👋 I encountered the same problem, how did you handle it?

@forehalo
Copy link
Member

forehalo commented Feb 28, 2024

we are recommending you start the selfhost server with docker image distribution(canary/beta/stable) the same as branch(canary/beta/stable) where your compose file located at.

there always some difference in between the compose file and docker image for different branches.

for example, if you are in affine repo at stable branch, you should make sure the compose file are using the image with stable tag. and the same for canary or beta branch.

we've already updated the docs to tell cloning with stable branch:

git clone https://github.com/toeverything/AFFiNE.git --branch stable

BTW, don't forget to pull the latest images

@fabiencharrasse
Copy link

Hello, with the last update i can't login to the "connect to AFFINE Cloud" form, i'm redirect every time to xxx:3010/signin?callbackUrl=%2Fauth%2FsignUp
I'm suppose to be able to use AFFINE_ADMIN_EMAIL and AFFINE_ADMIN_PASSWORD in this form, right ?

@EYHN EYHN closed this as completed Feb 28, 2024
@omarcostahamido
Copy link

@yguo
Copy link

yguo commented Feb 29, 2024

I encounter the following errors when using docker compose to do the self-hosting deployment.

"Error: Command failed: yarn predeploy
affine_selfhosted | at checkExecSyncError (node:child_process:890:11)
affine_selfhosted | at execSync (node:child_process:962:15)
affine_selfhosted | at runPredeployScript (file:///app/scripts/self-host-predeploy.js:44:3)
affine_selfhosted | at file:///app/scripts/self-host-predeploy.js:52:1
affine_selfhosted | at ModuleJob.run (node:internal/modules/esm/module_job:195:25)
affine_selfhosted | at async ModuleLoader.import (node:internal/modules/esm/loader:336:24)
affine_selfhosted | at async loadESM (node:internal/process/esm_loader:34:7)
affine_selfhosted | at async handleMainPromise (node:internal/modules/run_main:106:12) {
affine_selfhosted | status: 139,"

the branch I clone is 'stable' and the image is also tagged 'stable'

@WarlockFish
Copy link

image loading fail

图片

@endquote
Copy link

endquote commented Mar 1, 2024

I was able to get it running using the provided instructions.

Is it limited to just the admin user for now? Can I open it to registration for other users, and share pages between users?

@doldremus
Copy link

@endquote
I think the best option now is to connect the smtp service.
Example compose.yml environment parameters:

- MAILER_HOST=smtp.gmail.com
- MAILER_PORT=465
- MAILER_USER=noreply@gmail.com
- MAILER_PASSWORD=54745trurt5ttu
- MAILER_SENDER=AFFiNE <noreply@domain.com>
- MAILER_SECURE=true

@PhilippMolitor
Copy link

@acranglesim the env setting SERVER_FLAVOR=selfhosted is no longer valid. Omitting it now yields the correct configuration.
@fabiencharrasse I am facing the same issue as you. Each time I enter the email address for login, I get redirected to the same login form over and over again.

My Affine server logs show the following:

affine_1    | [Nest] 121  - 03/03/2024, 11:12:31 AM    WARN [NextAuth] NEXTAUTH_URL

Maybe NextAuth doesn't pick up the correct url from the affine server env config?

@Kerber0ss
Copy link

@PhilippMolitor same issue with login via email-address.

@omarcostahamido
Copy link

@EYHN consider reopening and help users with their issue?

@markoott
Copy link

markoott commented Mar 3, 2024

@PhilippMolitor @Kerber0ss
I initially encountered the same error. The reason for this was that I used a different e-mail address + password for my first configuration. When I recreated the Docker stack, the database and app were probably not recreated. I therefore deleted all folders first and rebuilt them. I was then able to log in with the admin account without any problems.

@Kerber0ss
Copy link

@markoott
I recreated stack several times. I try stable, beta, canary, removed old containers, images, install in other instance, but nothing, when type email - and press button - page reload and email again.

@endquote
Copy link

endquote commented Mar 4, 2024

@endquote I think the best option now is to connect the smtp service. Example compose.yml environment parameters:

- MAILER_HOST=smtp.gmail.com
- MAILER_PORT=465
- MAILER_USER=noreply@gmail.com
- MAILER_PASSWORD=54745trurt5ttu
- MAILER_SENDER=AFFiNE <noreply@domain.com>
- MAILER_SECURE=true

Thanks @doldremus. I set up these env vars and tried to invite someone else to a workspace. I got this error:

[Nest] 117  - 03/04/2024, 9:01:37 PM    WARN ;5;3m[WorkspaceResolver] failed to send 4eb2131e-e310-4afd-8081-1e1053d34326 invite email to [removed], but successfully revoked permission: Error: Mailer service is not configured.

Here are my settings:

MAILER_HOST=smtp.mycompany.com
MAILER_PASSWORD= # my server does not require a password
MAILER_PORT=587
MAILER_SECURE=true
MAILER_SENDER=AFFiNE <ux-noreply@mycompany.com>
MAILER_USER=ux-noreply@mycompany.com

I've had issues with this no-password thing in other apps, I wonder if that's the issue here.

I tried entering an email into the registration form, but I did not see the same error in the log. Maybe it only happens once.

I could make this a new issue somewhere else since this thread is getting mixed.

@EYHN
Copy link
Member Author

EYHN commented Mar 5, 2024

Please open a new issue if you need help

@fabiencharrasse
Copy link

For those who have problems with the login form, I posted my solution here: #6007 (comment)

@toeverything toeverything locked as too heated and limited conversation to collaborators Mar 5, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
type:epic Big feature type:feat Non-bug feature improvements
Projects
Archived in project
Development

No branches or pull requests