WAVES

Benchmarking the Robustness of Image Watermarks

Bang An^*¹, Mucong Ding^*¹, Tahseen Rabbani^*¹, Aakriti Agrawal¹, Yuancheng Xu¹, Chenghao Deng¹, Sicheng Zhu¹, Abdirisak Mohamed¹², Yuxin Wen¹, Tom Goldstein¹, Furong Huang¹,

¹University of Maryland, College Park
²SAP Labs, LLC.
^*Indicates Equal Contribution. First authors are ordered alphabetically.

Arxiv Github

🤗

HuggingFace

🏆

Leaderboard

🔮

Visualize

🌐

Twitter

Abstract

This paper investigates the weaknesses of image watermarking techniques. We present WAVES (Watermark Analysis via Enhanced Stress-testing), a novel benchmark for assessing watermark robustness, overcoming the limitations of current evaluation methods. WAVES integrates detection and identification tasks and establishes a standardized evaluation protocol comprised of a diverse range of stress tests. The attacks in WAVES range from traditional image distortions to advanced and novel variations of diffusive, and adversarial attacks. We introduce a normalized score of image quality degradation incorporating several widely used quality metrics. Our evaluation encompasses two key aspects: quality degradation and watermark detection performance after attacks. We benchmark watermarks and rank attacks based on the Performance vs. Quality 2D plots. Our comprehensive evaluation reveals previously undetected vulnerabilities of several modern watermarking algorithms. We envision WAVES as a toolkit for the future development of robust watermarking systems.

Workflow and Evaluation

What is watermark robustness? An AI company provides two services: (1) generate watermarked images, i.e., embed invisible messages, either through a post-processing method or an in-processing method, and (2) detect these messages when shown any of their watermarked images. There is an attack stage between the watermarking and detection stages. The watermarked images may experience natural distortions (e.g., compression, re-scaling) or manipulated by malicious users attempting to remove the watermarks. A robust watermarking method should still be able to detect the original message after an attack. WAVES can evaluate the robustness of any watermarks.

Evaluation of a single attack on a watermarking method. We first attack watermarked images over a variety of strengths (also labeled 'stg'). Then, we evaluate the detection performance (TPR@0.1%FPR) and a collection of image quality metrics such as PSNR, and plot a set of performance vs. quality plots. By normalizing and aggregating these quality metrics, we derive a consolidated 2D plot that represents the overall performance vs. quality for the evaluation.

Benchmarking watermarks and attacks. For each watermarking method, we plot all attacks on a unified performance vs. quality 2D plot to facilitate a detailed comparison. Based on this, we provide two additional analytical perspectives. We compare watermarks' robustness through the averaged performance under different attacks. We evaluate attacks' potency by ranking the quality at a specific performance threshold.

Benchmark Watermarks

Watermark detection performance (i.e., TPR@0.1%FPR) of Stable Signature, StegaStamp, and Tree-Ring watermarks after attacks via WAVES. We compute the Average TPR@0.1%FPR across all strength levels and further averaged this metric across different attacks and datasets. Lower Average TPR@0.1%FPR indicates higher vulnerability of the watermark to a certain type of attack. Right figure shows the distribution of quality degradation for each type of attack. Lower quality degradation is preferred.

User identification performance (i.e., accuracy) of Stable Signature, StegaStamp, and Tree-Ring watermarks after attacks via WAVES. We simulate identification tasks with total user counts of 100, 1,000, and 1000000.

Leaderboard for Watermark Detection

Attack	Tree-Ring						Stable Signature						StegaStamp

	Rank	Q@0.7P	Q@0.4P	Avg P	Avg Q	Rank	Q@0.7P	Q@0.4P	Avg P	Avg Q	Rank	Q@0.7P	Q@0.4P	Avg P	Avg Q
Dist-Rotation	8	-inf	0.434	0.131	0.648	12	0.613	0.642	0.400	0.650	4	0.454	0.500	0.288	0.616
Dist-RCrop	11	-inf	0.592	0.094	0.463	24	inf	inf	0.972	0.461	6	0.602	0.602	0.494	0.451
Dist-Erase	26	inf	inf	0.986	0.490	25	inf	inf	0.988	0.489	25	inf	inf	1.000	0.483
Dist-Bright	22	inf	inf	0.913	0.304	23	inf	inf	0.982	0.305	22	inf	inf	0.995	0.317
Dist-Contrast	23	inf	inf	0.949	0.243	20	inf	inf	0.979	0.243	17	inf	inf	0.994	0.231
Dist-Blur	21	1.105	1.437	0.551	1.221	5	-inf	-inf	0.000	1.204	9	0.897	0.970	0.280	1.198
Dist-Noise	16	0.427	inf	0.728	0.395	8	0.415	0.480	0.633	0.390	24	inf	inf	1.000	0.360
Dist-JPEG	17	0.499	0.499	0.700	0.284	9	0.485	0.485	0.540	0.284	21	inf	inf	0.995	0.263
DistCom-Geo	9	-inf	0.559	0.105	0.768	13	0.788	0.835	0.519	0.767	7	0.676	0.717	0.359	0.733
DistCom-Photo	23	inf	inf	0.947	0.242	20	inf	inf	0.981	0.243	17	inf	inf	0.994	0.239
DistCom-Deg	18	0.556	0.864	0.570	0.694	7	0.216	0.281	0.183	0.679	8	0.870	0.957	0.737	0.664
DistCom-All	10	-inf	0.575	0.123	0.908	11	0.550	0.623	0.176	0.900	10	0.995	1.096	0.682	0.870
Regen-Diff	6	-inf	0.307	0.258	0.323	1	-inf	-inf	0.000	0.300	2	0.333	inf	0.766	0.327
Regen-DiffP	6	-inf	0.308	0.256	0.327	1	-inf	-inf	0.000	0.303	1	0.336	0.356	0.763	0.329
Regen-VAE	19	0.578	0.578	0.701	0.348	10	0.545	0.545	0.340	0.339	23	inf	inf	1.000	0.343
Regen-KLVAE	14	0.257	inf	0.810	0.233	6	-inf	-inf	0.047	0.206	17	inf	inf	0.999	0.240
Rinse-2xDiff	5	-inf	0.270	0.220	0.357	3	-inf	-inf	0.000	0.332	3	0.390	0.402	0.778	0.366
Rinse-4xDiff	1	-inf	-inf	0.110	0.466	4	-inf	-inf	0.000	0.438	5	0.488	0.676	0.687	0.477
AdvEmbG-KLVAE8	4	-inf	0.168	0.259	0.253	20	inf	inf	0.985	0.249	17	inf	inf	1.000	0.232
AdvEmbB-RN18	15	0.288	inf	0.811	0.218	17	inf	inf	0.990	0.212	14	inf	inf	1.000	0.196
AdvEmbB-CLIP	20	0.697	inf	0.798	0.549	26	inf	inf	0.991	0.541	25	inf	inf	1.000	0.488
AdvEmbB-KLVAE16	12	0.158	0.309	0.540	0.238	19	inf	inf	0.983	0.233	14	inf	inf	1.000	0.206
AdvEmbB-SdxlVAE	13	0.214	inf	0.692	0.221	17	inf	inf	0.986	0.219	14	inf	inf	1.000	0.204
AdvCls-UnWM&WM	2	-inf	0.123	0.352	0.145	14	inf	inf	0.991	0.101	11	inf	inf	1.000	0.101
AdvCls-Real&WM	25	inf	inf	0.986	0.047	14	inf	inf	0.990	0.092	11	inf	inf	1.000	0.106
AdvCls-WM1&WM2	2	-inf	0.118	0.343	0.139	14	inf	inf	0.991	0.084	13	inf	inf	1.000	0.129

Leaderboard for User Identification

Attack	Tree-Ring						Stable Signature						StegaStamp

	Rank	Q@0.7P	Q@0.4P	Avg P	Avg Q	Rank	Q@0.7P	Q@0.4P	Avg P	Avg Q	Rank	Q@0.7P	Q@0.4P	Avg P	Avg Q
Dist-Rotation	8	-inf	0.434	0.131	0.648	12	0.613	0.642	0.400	0.650	4	0.454	0.500	0.288	0.616
Dist-RCrop	11	-inf	0.592	0.094	0.463	24	inf	inf	0.972	0.461	6	0.602	0.602	0.494	0.451
Dist-Erase	26	inf	inf	0.986	0.490	25	inf	inf	0.988	0.489	25	inf	inf	1.000	0.483
Dist-Bright	22	inf	inf	0.913	0.304	23	inf	inf	0.982	0.305	22	inf	inf	0.995	0.317
Dist-Contrast	23	inf	inf	0.949	0.243	20	inf	inf	0.979	0.243	17	inf	inf	0.994	0.231
Dist-Blur	21	1.105	1.437	0.551	1.221	5	-inf	-inf	0.000	1.204	9	0.897	0.970	0.280	1.198
Dist-Noise	16	0.427	inf	0.728	0.395	8	0.415	0.480	0.633	0.390	24	inf	inf	1.000	0.360
Dist-JPEG	17	0.499	0.499	0.700	0.284	9	0.485	0.485	0.540	0.284	21	inf	inf	0.995	0.263
DistCom-Geo	9	-inf	0.559	0.105	0.768	13	0.788	0.835	0.519	0.767	7	0.676	0.717	0.359	0.733
DistCom-Photo	23	inf	inf	0.947	0.242	20	inf	inf	0.981	0.243	17	inf	inf	0.994	0.239
DistCom-Deg	18	0.556	0.864	0.570	0.694	7	0.216	0.281	0.183	0.679	8	0.870	0.957	0.737	0.664
DistCom-All	10	-inf	0.575	0.123	0.908	11	0.550	0.623	0.176	0.900	10	0.995	1.096	0.682	0.870
Regen-Diff	6	-inf	0.307	0.258	0.323	1	-inf	-inf	0.000	0.300	2	0.333	inf	0.766	0.327
Regen-DiffP	6	-inf	0.308	0.256	0.327	1	-inf	-inf	0.000	0.303	1	0.336	0.356	0.763	0.329
Regen-VAE	19	0.578	0.578	0.701	0.348	10	0.545	0.545	0.340	0.339	23	inf	inf	1.000	0.343
Regen-KLVAE	14	0.257	inf	0.810	0.233	6	-inf	-inf	0.047	0.206	17	inf	inf	0.999	0.240
Rinse-2xDiff	5	-inf	0.270	0.220	0.357	3	-inf	-inf	0.000	0.332	3	0.390	0.402	0.778	0.366
Rinse-4xDiff	1	-inf	-inf	0.110	0.466	4	-inf	-inf	0.000	0.438	5	0.488	0.676	0.687	0.477
AdvEmbG-KLVAE8	4	-inf	0.168	0.259	0.253	20	inf	inf	0.985	0.249	17	inf	inf	1.000	0.232
AdvEmbB-RN18	15	0.288	inf	0.811	0.218	17	inf	inf	0.990	0.212	14	inf	inf	1.000	0.196
AdvEmbB-CLIP	20	0.697	inf	0.798	0.549	26	inf	inf	0.991	0.541	25	inf	inf	1.000	0.488
AdvEmbB-KLVAE16	12	0.158	0.309	0.540	0.238	19	inf	inf	0.983	0.233	14	inf	inf	1.000	0.206
AdvEmbB-SdxlVAE	13	0.214	inf	0.692	0.221	17	inf	inf	0.986	0.219	14	inf	inf	1.000	0.204
AdvCls-UnWM&WM	2	-inf	0.123	0.352	0.145	14	inf	inf	0.991	0.101	11	inf	inf	1.000	0.101
AdvCls-Real&WM	25	inf	inf	0.986	0.047	14	inf	inf	0.990	0.092	11	inf	inf	1.000	0.106
AdvCls-WM1&WM2	2	-inf	0.118	0.343	0.139	14	inf	inf	0.991	0.084	13	inf	inf	1.000	0.129

Performance vs. Quality Plots

Distortions.

Regeneration Attacks.

Adversarial Attacks.

BibTeX

@misc{an2024benchmarking,
        title={Benchmarking the Robustness of Image Watermarks}, 
        author={Bang An and Mucong Ding and Tahseen Rabbani and Aakriti Agrawal and Yuancheng Xu and Chenghao Deng and Sicheng Zhu and Abdirisak Mohamed and Yuxin Wen and Tom Goldstein and Furong Huang},
        year={2024},
        eprint={2401.08573},
        archivePrefix={arXiv},
        primaryClass={cs.CV}
  }