Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Contribute to GitLab
Sign in / Register
Toggle navigation
S
stable-diffusion-webui
Project
Project
Details
Activity
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
Administrator
stable-diffusion-webui
Commits
a26f157a
Unverified
Commit
a26f157a
authored
Sep 12, 2022
by
AUTOMATIC1111
Committed by
GitHub
Sep 12, 2022
Browse files
Options
Browse Files
Download
Plain Diff
Merge pull request #335 from C43H66N12O12S2/attention-update
Update cross attention to the newest version
parents
834b6e39
aaea8b44
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
4 additions
and
3 deletions
+4
-3
sd_hijack.py
modules/sd_hijack.py
+4
-3
No files found.
modules/sd_hijack.py
View file @
a26f157a
...
...
@@ -67,8 +67,9 @@ def split_cross_attention_forward(self, x, context=None, mask=None):
mem_free_total
=
mem_free_cuda
+
mem_free_torch
gb
=
1024
**
3
tensor_size
=
q
.
shape
[
0
]
*
q
.
shape
[
1
]
*
k
.
shape
[
1
]
*
4
mem_required
=
tensor_size
*
2.5
tensor_size
=
q
.
shape
[
0
]
*
q
.
shape
[
1
]
*
k
.
shape
[
1
]
*
q
.
element_size
()
modifier
=
3
if
q
.
element_size
()
==
2
else
2.5
mem_required
=
tensor_size
*
modifier
steps
=
1
if
mem_required
>
mem_free_total
:
...
...
@@ -86,7 +87,7 @@ def split_cross_attention_forward(self, x, context=None, mask=None):
end
=
i
+
slice_size
s1
=
einsum
(
'b i d, b j d -> b i j'
,
q
[:,
i
:
end
],
k
)
*
self
.
scale
s2
=
s1
.
softmax
(
dim
=-
1
)
s2
=
s1
.
softmax
(
dim
=-
1
,
dtype
=
q
.
dtype
)
del
s1
r1
[:,
i
:
end
]
=
einsum
(
'b i j, b j d -> b i d'
,
s2
,
v
)
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment