{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":219035799,"defaultBranch":"main","name":"tokenizers","ownerLogin":"huggingface","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2019-11-01T17:52:20.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/25720743?v=4","public":true,"private":false,"isOrgOwned":true},"refInfo":{"name":"","listCacheKey":"v0:1714988979.0","currentOid":""},"activityList":{"items":[{"before":"4992d242a5f4793211a83646ee479b9eb8d6b705","after":null,"ref":"refs/heads/break_decoder_added_tokens","pushedAt":"2024-05-06T09:49:39.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"Narsil","name":"Nicolas Patry","path":"/Narsil","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/204321?s=80&v=4"}},{"before":"f2ec3b239b0a7a9866b01ec5cbd4d44243a40a16","after":"25aee8b88c8de3c5a52e2f9cb6281d6df00ad516","ref":"refs/heads/main","pushedAt":"2024-05-06T09:49:38.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"Narsil","name":"Nicolas Patry","path":"/Narsil","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/204321?s=80&v=4"},"commit":{"message":"[BREAKING CHANGE] Ignore added_tokens (both special and not) in the decoder (#1513)\n\n* [BREAKING CHANGE] Ignore added_tokens (both special and not) in the\r\ndecoder\r\n\r\nCauses issues with `ByteLevel` messing up some `AddedTokens` with some\r\nutf-8 range used in the bytelevel mapping.\r\n\r\nThis commit tests the extend of the damage of ignoring the decoder for\r\nthose tokens.\r\n\r\n* Format.\r\n\r\n* Installing cargo audit.\r\n\r\n* Minor fix.\r\n\r\n* Fixing \"bug\" in node/python.\r\n\r\n* Autoformat.\r\n\r\n* Clippy.\r\n\r\n* Only prefix space when there's no decoder.","shortMessageHtmlLink":"[BREAKING CHANGE] Ignore added_tokens (both special and not) in the d…"}},{"before":"443e5333d6dcea7f8dda5859d435c0994fca831b","after":null,"ref":"refs/heads/dependabot/npm_and_yarn/tokenizers/examples/unstable_wasm/www/express-4.19.2","pushedAt":"2024-05-04T01:49:10.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"dependabot[bot]","name":null,"path":"/apps/dependabot","primaryAvatarUrl":"https://avatars.githubusercontent.com/in/29110?s=80&v=4"}},{"before":"71c2a8d01a56cd7bd28148c309e210c47dac78e7","after":"f2ec3b239b0a7a9866b01ec5cbd4d44243a40a16","ref":"refs/heads/main","pushedAt":"2024-04-30T13:53:47.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ArthurZucker","name":"Arthur","path":"/ArthurZucker","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48595927?s=80&v=4"},"commit":{"message":"remove enforcement of non special when adding tokens (#1521)\n\n* remove enforcement of non special when adding tokens\r\n\r\n* mut no longer needed\r\n\r\n* add a small test\r\n\r\n* nit\r\n\r\n* style\r\n\r\n* audit\r\n\r\n* ignore cargo audit's own vulnerability\r\n\r\n* update\r\n\r\n* revert\r\n\r\n* remove CVE","shortMessageHtmlLink":"remove enforcement of non special when adding tokens (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2270901030\" data-permission-text=\"Title is private\" data-url=\"https://github.com/huggingface/tokenizers/issues/1521\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/huggingface/tokenizers/pull/1521/hovercard\" href=\"https://github.com/huggingface/tokenizers/pull/1521\">#1521</a>)"}},{"before":"3579eb61112cd6c89d6a9e72ae97b3dd853ddf1f","after":null,"ref":"refs/heads/fix-add-tokens","pushedAt":"2024-04-30T13:53:47.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"ArthurZucker","name":"Arthur","path":"/ArthurZucker","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48595927?s=80&v=4"}},{"before":"7b8d4e68398fa11669eb39c964b85422b8dc7398","after":"3579eb61112cd6c89d6a9e72ae97b3dd853ddf1f","ref":"refs/heads/fix-add-tokens","pushedAt":"2024-04-30T13:46:02.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ArthurZucker","name":"Arthur","path":"/ArthurZucker","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48595927?s=80&v=4"},"commit":{"message":"remove CVE","shortMessageHtmlLink":"remove CVE"}},{"before":"d70ba4236c806fff002d45aa3c42ec06004cd9bf","after":"7b8d4e68398fa11669eb39c964b85422b8dc7398","ref":"refs/heads/fix-add-tokens","pushedAt":"2024-04-30T13:45:26.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ArthurZucker","name":"Arthur","path":"/ArthurZucker","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48595927?s=80&v=4"},"commit":{"message":"revert","shortMessageHtmlLink":"revert"}},{"before":"65b495895ff411e7a208a368796a11624771bfaf","after":"d70ba4236c806fff002d45aa3c42ec06004cd9bf","ref":"refs/heads/fix-add-tokens","pushedAt":"2024-04-30T13:28:10.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ArthurZucker","name":"Arthur","path":"/ArthurZucker","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48595927?s=80&v=4"},"commit":{"message":"update","shortMessageHtmlLink":"update"}},{"before":"230989610635fc40d1237f078600428bd6937727","after":"65b495895ff411e7a208a368796a11624771bfaf","ref":"refs/heads/fix-add-tokens","pushedAt":"2024-04-30T13:23:46.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ArthurZucker","name":"Arthur","path":"/ArthurZucker","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48595927?s=80&v=4"},"commit":{"message":"ignore cargo audit's own vulnerability","shortMessageHtmlLink":"ignore cargo audit's own vulnerability"}},{"before":"e3fec4f7cbd8a72ad614699999f55d6fbd188785","after":"230989610635fc40d1237f078600428bd6937727","ref":"refs/heads/fix-add-tokens","pushedAt":"2024-04-30T12:55:24.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ArthurZucker","name":"Arthur","path":"/ArthurZucker","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48595927?s=80&v=4"},"commit":{"message":"audit","shortMessageHtmlLink":"audit"}},{"before":"c03582d5c5bb3b7a788580a547a8e243549ec19c","after":"e3fec4f7cbd8a72ad614699999f55d6fbd188785","ref":"refs/heads/fix-add-tokens","pushedAt":"2024-04-30T12:18:19.000Z","pushType":"push","commitsCount":3,"pusher":{"login":"ArthurZucker","name":"Arthur","path":"/ArthurZucker","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48595927?s=80&v=4"},"commit":{"message":"style","shortMessageHtmlLink":"style"}},{"before":"9cb07d252839aaca1d264ec54556f86961ad34fa","after":"c03582d5c5bb3b7a788580a547a8e243549ec19c","ref":"refs/heads/fix-add-tokens","pushedAt":"2024-04-30T09:14:58.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ArthurZucker","name":"Arthur","path":"/ArthurZucker","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48595927?s=80&v=4"},"commit":{"message":"mut no longer needed","shortMessageHtmlLink":"mut no longer needed"}},{"before":null,"after":"9cb07d252839aaca1d264ec54556f86961ad34fa","ref":"refs/heads/fix-add-tokens","pushedAt":"2024-04-30T08:53:30.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"ArthurZucker","name":"Arthur","path":"/ArthurZucker","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48595927?s=80&v=4"},"commit":{"message":"remove enforcement of non special when adding tokens","shortMessageHtmlLink":"remove enforcement of non special when adding tokens"}},{"before":"6682e5c6b82970ca5ed9e27c10ce1fc2c9f1f35e","after":"4992d242a5f4793211a83646ee479b9eb8d6b705","ref":"refs/heads/break_decoder_added_tokens","pushedAt":"2024-04-29T09:39:05.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Narsil","name":"Nicolas Patry","path":"/Narsil","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/204321?s=80&v=4"},"commit":{"message":"Only prefix space when there's no decoder.","shortMessageHtmlLink":"Only prefix space when there's no decoder."}},{"before":"58bff979275891765d9c5ab0e02e06b9dbf8ed87","after":null,"ref":"refs/heads/dependabot/npm_and_yarn/tokenizers/examples/unstable_wasm/www/webpack-dev-middleware-5.3.4","pushedAt":"2024-04-28T01:50:26.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"dependabot[bot]","name":null,"path":"/apps/dependabot","primaryAvatarUrl":"https://avatars.githubusercontent.com/in/29110?s=80&v=4"}},{"before":"53be059c6063c985d9452e36bb63281368d600ce","after":"6682e5c6b82970ca5ed9e27c10ce1fc2c9f1f35e","ref":"refs/heads/break_decoder_added_tokens","pushedAt":"2024-04-24T16:51:10.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Narsil","name":"Nicolas Patry","path":"/Narsil","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/204321?s=80&v=4"},"commit":{"message":"Clippy.","shortMessageHtmlLink":"Clippy."}},{"before":"cb928c046cef50c8cd66e74ea6b3dc6fbc6e9f80","after":"53be059c6063c985d9452e36bb63281368d600ce","ref":"refs/heads/break_decoder_added_tokens","pushedAt":"2024-04-24T16:47:54.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Narsil","name":"Nicolas Patry","path":"/Narsil","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/204321?s=80&v=4"},"commit":{"message":"Autoformat.","shortMessageHtmlLink":"Autoformat."}},{"before":"44f6a84d8a93a78007d901cbfc7ee372f4f7c10f","after":"cb928c046cef50c8cd66e74ea6b3dc6fbc6e9f80","ref":"refs/heads/break_decoder_added_tokens","pushedAt":"2024-04-24T16:26:07.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Narsil","name":"Nicolas Patry","path":"/Narsil","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/204321?s=80&v=4"},"commit":{"message":"Fixing \"bug\" in node/python.","shortMessageHtmlLink":"Fixing \"bug\" in node/python."}},{"before":"5b684db922de8fe498b2138759b2f3699b620698","after":"44f6a84d8a93a78007d901cbfc7ee372f4f7c10f","ref":"refs/heads/break_decoder_added_tokens","pushedAt":"2024-04-24T16:18:37.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Narsil","name":"Nicolas Patry","path":"/Narsil","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/204321?s=80&v=4"},"commit":{"message":"Minor fix.","shortMessageHtmlLink":"Minor fix."}},{"before":"ed9eb5abed837c7a8292593c559e9d0d9911255d","after":"5b684db922de8fe498b2138759b2f3699b620698","ref":"refs/heads/break_decoder_added_tokens","pushedAt":"2024-04-24T16:10:42.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Narsil","name":"Nicolas Patry","path":"/Narsil","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/204321?s=80&v=4"},"commit":{"message":"Installing cargo audit.","shortMessageHtmlLink":"Installing cargo audit."}},{"before":"a5d4584653ed8e87f9f8dd0a5168b2df570961e9","after":"ed9eb5abed837c7a8292593c559e9d0d9911255d","ref":"refs/heads/break_decoder_added_tokens","pushedAt":"2024-04-24T16:10:03.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Narsil","name":"Nicolas Patry","path":"/Narsil","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/204321?s=80&v=4"},"commit":{"message":"Format.","shortMessageHtmlLink":"Format."}},{"before":null,"after":"a5d4584653ed8e87f9f8dd0a5168b2df570961e9","ref":"refs/heads/break_decoder_added_tokens","pushedAt":"2024-04-24T15:50:47.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"Narsil","name":"Nicolas Patry","path":"/Narsil","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/204321?s=80&v=4"},"commit":{"message":"[BREAKING CHANGE] Ignore added_tokens (both special and not) in the\ndecoder\n\nCauses issues with `ByteLevel` messing up some `AddedTokens` with some\nutf-8 range used in the bytelevel mapping.\n\nThis commit tests the extend of the damage of ignoring the decoder for\nthose tokens.","shortMessageHtmlLink":"[BREAKING CHANGE] Ignore added_tokens (both special and not) in the"}},{"before":"b5ba6d12bd2ac5e354cf8aa87e16e91a7d9427cc","after":null,"ref":"refs/heads/dependabot/npm_and_yarn/tokenizers/examples/unstable_wasm/www/follow-redirects-1.15.6","pushedAt":"2024-04-21T01:50:08.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"dependabot[bot]","name":null,"path":"/apps/dependabot","primaryAvatarUrl":"https://avatars.githubusercontent.com/in/29110?s=80&v=4"}},{"before":"7733bc25d62dedcd1b4f2bd59b761720bf19c284","after":"71c2a8d01a56cd7bd28148c309e210c47dac78e7","ref":"refs/heads/main","pushedAt":"2024-04-17T21:17:23.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ArthurZucker","name":"Arthur","path":"/ArthurZucker","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48595927?s=80&v=4"},"commit":{"message":"update dev version so 0.19.1","shortMessageHtmlLink":"update dev version so 0.19.1"}},{"before":"da307684adb81fe5ed7d340d139d669aeb3ca1ba","after":"3b3c9604f951850be398e45e4a569242fb2319c1","ref":"refs/heads/branch_0.19.1","pushedAt":"2024-04-17T21:09:10.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ArthurZucker","name":"Arthur","path":"/ArthurZucker","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48595927?s=80&v=4"},"commit":{"message":"version 0.19.1","shortMessageHtmlLink":"version 0.19.1"}},{"before":null,"after":"da307684adb81fe5ed7d340d139d669aeb3ca1ba","ref":"refs/heads/branch_0.19.1","pushedAt":"2024-04-17T20:07:00.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"ArthurZucker","name":"Arthur","path":"/ArthurZucker","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48595927?s=80&v=4"},"commit":{"message":"\"0.19.1-rc0\" and Cargo.lock","shortMessageHtmlLink":"\"0.19.1-rc0\" and Cargo.lock"}},{"before":"e1807b7137e7fef9a81db1e4cef5ebeefe7c631f","after":null,"ref":"refs/heads/fix-serialize","pushedAt":"2024-04-17T19:56:49.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"Narsil","name":"Nicolas Patry","path":"/Narsil","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/204321?s=80&v=4"}},{"before":"91393ef75e3d6075a689a2898a5a844b369d1b09","after":"7733bc25d62dedcd1b4f2bd59b761720bf19c284","ref":"refs/heads/main","pushedAt":"2024-04-17T19:56:48.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"Narsil","name":"Nicolas Patry","path":"/Narsil","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/204321?s=80&v=4"},"commit":{"message":"add serialization for `ignore_merges` (#1504)\n\n* add serialization for `ignore_merges`\r\n\r\n* add serialization tests\r\n\r\n* deserialize without `ignore_merges`","shortMessageHtmlLink":"add serialization for <code>ignore_merges</code> (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2249050411\" data-permission-text=\"Title is private\" data-url=\"https://github.com/huggingface/tokenizers/issues/1504\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/huggingface/tokenizers/pull/1504/hovercard\" href=\"https://github.com/huggingface/tokenizers/pull/1504\">#1504</a>)"}},{"before":"cfb8ddf5a84b1f4ebdd5a4e88c5eea6f729c96b9","after":"e1807b7137e7fef9a81db1e4cef5ebeefe7c631f","ref":"refs/heads/fix-serialize","pushedAt":"2024-04-17T19:39:51.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ArthurZucker","name":"Arthur","path":"/ArthurZucker","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48595927?s=80&v=4"},"commit":{"message":"deserialize without `ignore_merges`","shortMessageHtmlLink":"deserialize without <code>ignore_merges</code>"}},{"before":"69538d2df399446bd45c1a7071526bd1729fd40b","after":"cfb8ddf5a84b1f4ebdd5a4e88c5eea6f729c96b9","ref":"refs/heads/fix-serialize","pushedAt":"2024-04-17T19:36:34.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ArthurZucker","name":"Arthur","path":"/ArthurZucker","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48595927?s=80&v=4"},"commit":{"message":"add serialization tests","shortMessageHtmlLink":"add serialization tests"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAEQl_fRwA","startCursor":null,"endCursor":null}},"title":"Activity · huggingface/tokenizers"}